Re: Build a menu for an HTML publish

[Bruno Barbier]

Hi Sébastien,

Sébastien Gendre  writes:

[...]
> I cannot found a way to define "very-strawbery" value in an Org-mode
> buffer and having it used while I manually export the Org-mode buffer.
>
> I tried with "#+very-strawbery: test123" and "#+OPTIONS: very-strawbery:
> test123", without success.
>
> Someone have an idea ?

IIUC, "BIND" should work.

You put this in the exported file:

#+BIND: a-string "3"
#+BIND: a-number 3

And you can then use these variables directly:

(defun my/preamble-test (info)
   (message "My string: %S" a-string)
   (message "My number: %S" a-number))

This works only if 'org-export-allow-bind-keywords' is t; else Org
ignores the "#+BIND" instructions.

HTH,

Bruno

Re: Pending contents in org documents

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
[...]
>>> Ideally, we should have no hard-coded color names.
>>
[...]
> If you have no ideas about faces to inherit from, better keep hard-coded
> colors.
>
> (Also, this is not too critical; just something nice to have for better
> inter-operability with Emacs themes)

I've simplified everything, inheriting only basic faces that come
preloaded with emacs.  The display looks good enough, and, the code
will be easier to maintain.  Thanks.



>
>>>> (cl-defun org-pending--update (reglock status data)
>>>
>>> No docstring.  Please, add.
>>
>> I added some basic documentation; I hope this is enough (this is an
>> internal function).
>
> I prefer to have docstrings for every function, including internal
> functions. This will make life easier for future contributors when
> diagnosing whether a given function behaves as it supposed to.
>
> Ideally, we should detail what the function is expected to do by its
> callers in its docstring.  This way, we have a way to check if the code
> behaves as it should in the future, after situations like external API
> change or unexpected change in another internal API.

Ideally, I agree :)

Though, IME, for internal details of experimental code, improving the
code readability and adding inline comments is more effective.



>>   (let ((map (make-sparse-keymap)))
>> (dolist (k `([touchscreen-down] [mouse-2] [mouse-1]))
>>   (define-key map k 'org-pending-describe-reglock-at-point))
>> (when read-only
>>   (define-key map [13] 'org-pending-describe-reglock-at-point))
>> map))
>
> Nitpick: #'org-pending... - this will help compiler to catch problems
> if something happens to `org-pending-describe-reglock-at-point' (like
> rename).

Done.



> Also, [13] is not very readable.  Better use ?\C-m (or what did you mean?)

I just blindly pasted those keys from the Emacs source code, to get
the button-like behavior.  I've replaced it, using keymap-set and
"RET".

To me, '?\C-m' doesn't look simpler than the ASCII code 13 :)



>> (defun org-pending--make-overlay (type beg-end)
>> ...
>>   (let ((overlay (make-overlay (car beg-end) (cdr beg-end)))
>> (read-only
>>  (list
>>   (lambda (&rest _)
>> (signal 'org-pending-error
>> (list "Cannot modify a region containing pending 
>> content"))
>
> You can factor this lambda out into an internal constant.

Done.


>> (defun org-pending--make-overlay (type beg-end)
>>   "Create a pending overlay of type TYPE between BEG-END.
>
>> The pair BEG-END contains 2 positions (BEG . END).
>> Create an overlay between BEGIN and END.  Return it.
>
>> See `org-pending--delete-overlay' to delete it."
>
> It would be nice to have the docstring detail what kinds of properties
> are applied to the overlay: (1) that it is read-only; (2)
> org-pending--owner; (3) org-pending--before-delete; (4) keymap.

I tried to improve the current documentation (only ':region' overlays
are read-only). Almost every line of code is now in the documentation:
that internal function is now officially declared "carved in stone",
and, it should ship it as-is :)


I've renamed `org-pending--owner' to `org-pending--real-owner', and I
improved the comment about it in the code.  The property
`org-pending--before-delete' is now gone.


> It is especially important to document properties that other functions
> make use of.

That's why I like closures so much: keep the internal details internal
in one place; no need to split the logic and drop many parts in many
different places :)



>>   (let ((overlay (make-overlay (car beg-end) (cdr beg-end)))
>> (read-only
>>  (list
>>   (lambda (&rest _)
>> (signal 'org-pending-error
>> (list "Cannot modify a region containing pending 
>> content"))
>
> May you factor this lambda out into an internal constant?

I think it's the same as above, or I missed something.
Done ? :)


>
>> (cl-flet ((make-read-only (ovl)
>> "Make the overly OVL read-only."
>>(overlay-put ovl 'modification-hooks read-only)
>>(overlay-put ovl 'insert-in-front-hooks read-only)
>>(overlay-put ovl 'insert-behind-hooks read-only)))
>
> Or maybe even factor out make-read-only into an internal function that
> can mark/unmark overlay/region with read-only text properties

Re: Strange warnings from htmlize.el

[Bruno Barbier]

Hi Angelo,

Angelo Graziosi  writes:

> With the last Emacs master build (20240731_183641) I get a lot of 
> warnings like this:
>
>
> Warning (emacs): Cannot fontify source block (htmlize.el >= 1.34 required)
>
>
> when I export an old ORG document in HTML (C-c C-e h h).
>
> Until some time ago, I had  exported the same document and many more in 
> the same manner and never seen those warnings (just Emacs master and ORG 
> versions changed).
>
> What happened?

This warning is from Org. It's telling you that you need to install the
htmlize package in your new Emacs.

Bruno

Re: Pending contents in org documents

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>>>> I added a function 'org-pending-unlock-NOW!' which unlock the region
>>>> immediately.  The uppercase "NOW!" emphasizes that it's not the
>>>> "safe" way to unlock a region.
>>>
>>> I expect to see this function called by some kind of button in the
>>> details buffer, so that users can actually call it.
>>
>> This function should not be used, not even in code, except to work
>> around bugs.  I would prefer not to provide a button for it.
>
> Then, there is no point in this function - users will never know about
> it. Maybe you do expose it as a button, but also supply a yes/no prompt
> asking for confirmation?
>

The function 'org-pending-unlock-NOW!' is part of the API, it's not a
command Emacs end users.

If we make it a command and display that button by default, we'll need
also an option to not display it by default, and, probably an other
option to not ask for confirmation.  Let see later if we really need
to provide all this.


>>> We should provide a user option to suppress the query. Something like
>>> what `confirm-kill-processes' does. Maybe even support something akin
>>> `process-query-on-exit-flag', but for reglocks.
>>
>> IMHO, it's the package that uses org-pending that should provide such a
>> feature: I may not care about loosing data by ignoring on-going code
>> block executions, but, I'll probably care if Emacs aborts some money
>> transfers.
>
> I kindly disagree. I am not proposing anything dramatically different
> from what Emacs already does.
>
> If you look into `confirm-kill-process' docstring, you will see that
> Emacs does, in fact, have mechanisms for Elisp packages to tell whether a
> given process is important or not (`process-query-on-exit-flag').
>
> Yet, `confirm-kill-processes' overrides that mechanism if Emacs user
> wishes to and sets the value to non-default nil.
>
> There is nothing principally different in org-pending users compared to
> processes. So, in order to conform with the rest of Emacs API, we do
> need to provide an equivalent user customization.
> We will not suppress queries by default, but we ought to give users an
> option to suppress these queries. Maybe even follow
> `confirm-kill-process' setting in how org-pending behaves.

In my view, reglocks are more about regions that are being updated, than
processes; they are kind of _planed_ buffer modifications.

But, anyway, I added a user option
`org-pending-confirm-ignore-reglocks-on-exit' that allows to ignore
pending locks when exiting.


>>> Please describe what the default value of ON-OUTCOME (when ON-OUTCOME is
>>> not explicitly provided) does right in the docstring.
>>
>> Done.
>
> Thanks!
> There is one typo in the relevant commit:
>
> + The default ON-OUTCOME
> +function replaces the region on success and ignore failures; in all
> ^ignores
>
Thanks.


>> (defface org-pending-outcome-failure
>> ...
>> (defface org-pending-outcome-success
>
> If possible, please derive the faces from either org-faces.el or from
> built-in faces like the ones listed in `modus-themes-faces' variable.
>
> Ideally, we should have no hard-coded color names.

They were derived from built-in ones (error, success, org-tag, etc.).

I've redesigned the faces and put them all in org-pending, so that
org-pending is indenpendent of Org.

I'm now computing some colours from built-in faces to avoid colour
names. I'm not sure it's what you meant, as even Org itself doesn't do
this.


>> :version "30.1"
> Please, use :package-version instead of :version.

I'm now using:
   :package-version '(Org . "9.7")


>> (defun org-pending--status-still-pending-p (status)
>
> This function is trivial and used only once. Maybe you can just inline
> it.

Right.  But it also defines what "still pending" means.  I prefer to
keep it.


>
>> (defvar org-pending--outcome-keymap
>> (defvar org-pending--pending-keymap
>
> Why are these keymaps internal? May we better expose them to users for
> modification?

They are now part of the public API.



> It may also be a good idea to add some actual keyboard binding to this
> map.

Right.  I've improved the keymaps, trying to work like a button when
the text is read-only.



>> (defun org-pending--popup-failure-details (exc)
>
> This _internal_ function is unused in org-pending.el, but instead used
> in ob-core.el.  I am not yet looking into the changes outsid

Re: Pending contents in org documents

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>> I added a function 'org-pending-unlock-NOW!' which unlock the region
>> immediately.  The uppercase "NOW!" emphasizes that it's not the
>> "safe" way to unlock a region.
>
> I expect to see this function called by some kind of button in the
> details buffer, so that users can actually call it.

This function should not be used, not even in code, except to work
around bugs.  I would prefer not to provide a button for it.


> Also, a few more small comments on the commentary section:
>
>> ;; The library makes locks visible to the user using text properties
>> ;; and/or overlays.  It diplays and updates the status while the
> * displays

Thanks.


>> ;; If the user kills a buffer, or, kills Emacs, some locks may have to
>> ;; be destroyed.  The library will ask the user to confirm if an
>> ;; operation requires to destroy some locks.  See the field
>> ;; `before-destroy-function' of REGLOCK object, if you need to do
>> ;; something before a lock is destroyed.
>
> We should provide a user option to suppress the query. Something like
> what `confirm-kill-processes' does. Maybe even support something akin
> `process-query-on-exit-flag', but for reglocks.

IMHO, it's the package that uses org-pending that should provide such a
feature: I may not care about loosing data by ignoring on-going code
block executions, but, I'll probably care if Emacs aborts some money
transfers.


>
>> ... feel free to look at the docstrings of the
>> ;; cl-defstruct `org-pending-reglock' for the full documentation.
>
> Maybe better refer to M-x cl-describe-type org-pending-reglock. It will
> look nicer.

Thanks. I didn't know how to display that documentation actually :)

As this library is for developpers, I've used the syntax:
 (cl-describe-type 'org-pending-reglock)
so that the documentation is only one click away.


>> (cl-defun org-pending (region
>> ...
>> On receiving the outcome (a :success or :failure message, sent with
>> `org-pending-send-update'), remove the region protection.  Call
>> ON-OUTCOME with the reglock and the outcome, from the position from
>> where the REGLOCK was created.  If ON-OUTCOME returns a region (a
>> pair (start position . end position)), use it to report the
>> success/failure using visual hints on that region.  If ON-OUTCOME
>> returns nothing, don't display outcome marks.
>
> Please describe what the default value of ON-OUTCOME (when ON-OUTCOME is
> not explicitly provided) does right in the docstring.

Done.



>> You may set/update the following fields of your reglock to customize its
>> behavior:
>>- Emacs may have to destroy your locks; see the field
>>  `before-destroy-function' if you wish to do something before your
>>  lock is destroyed.
>>- The user may ask Emacs to cancel your lock; see the field
>>  `user-cancel-function' to override the default cancel function.
>>- The user may request a description of the lock; see the the field
>>  `insert-details-function' to add custom information when your
>>  lock is displayed to the user.
>> 
>> You may add/update your own properties to your reglock using the field
>> `properties', which is an association list.
>
> Here, we may also refer to cl-describe-type for more details about the fields.

Done.



>>   ( user-cancel-function nil
>> :documentation
>> "Function called when the user wish to cancel this REGLOCK,
>> with the REGLOCK as argument.  This function must return immediately; it
>> may, asynchronously, stop some processing and release resources; and,
>> once this is done, it should send the outcome to the REGLOCK (using
>> `org-pending-send-update', so that the region is unlocked and the
>> REGLOCK destroyed). The default value is
>> `org-pending--user-cancel-default'" )
>
> How come the default value is `org-pending--user-cancel-default' when it
> is nil?

The only way to build a lock is to use the function 'org-pending'; and
that function was setting it to that value.  I've moved the setting
to the defstruct definition.


>> (cl-defstruct (org-pending-reglock
>> ...
>
> It would be nice to define slot types for each slot.

I added that to my todo list: I'll have to learn how to write types in
elisp (or common lisp?) to be able to do that (... the very bottom of my
list :)
)


>>  insert-details-function nil
>> When non-nil, function called to insert custom details at the end of
>> ‘org-pendi

Re: Pending contents in org documents

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>> About how much to decorate, it depends on the user, I guess.  For
>> example, when org-pending is used for org babel, it should be obvious
>> that one has to click on "#+RESULTS:".
>>
>> The current decoration is not the best for discoverability, indeed.
>> But decorating the whole outcome region would be too much IMHO, and,
>> it might interfer with other buffer fontifications.
>
> We may do what flycheck/flyspell do. Maybe fontifying not the whole
> region, but at least the region part where the indication was placed.
>
> I really find the current behavior unintuitive even for experienced Emacs
> users.

I added 2 faces;  org-pending is now using those faces for the
whole outcome region.  I personally don't like it and won't use it
though :)


>> I've refactored the code and added two variables so that it's now
>> configurable, see 'org-pending-outcome-pre-display-function' and
>> 'org-pending-outcome-post-display-function'.
>
> Makes sense. Although, "display" part in the names is very confusing -
> it sounds way too similar to `pre-redisplay-functions', which is
> something entirely different.

I've renamed them to pre-insert-outcome/post-insert-outcome.


> What about removing org-pending-output-pre-display-function entirely (we
> can add it later, if necessary) and renaming
> org-pending-outcome-post-display-function to
> `org-pending-on-outcome-functions' - an abnormal hook executed after
> ON-OUTCOME action.

I am using it, So, I would prefer to keep it :)

The timing does matter, so I would prefer to keep being explicit about
it (i.e. keep the pre/post prefix).

These 2 functions define the implementation; they are not hooks.


>>> 2. I tried to do M-x undo while the reglock was active, and got an
>>>error. I'd expect that undo would work, skipping the region.
>>
>> I'm not sure exactly what you did, nor which error you got.  I've
>> noticed that the hacks (to handle indirect buffers) were flagging the
>> buffer as "modified" (using text properties).  I've fixed that.
>>
>> Is the problem solved now ?
>
> No.
>
> What I did is:
>
> 1. make repro
> 2. Insert the example code from the top comment and uncomment it
> 3. M-x eval-buffer
> 4. *While regloc is active*, press C-x u
> 5. Observe
>
> Debugger entered--Lisp error: (org-pending-error "Cannot modify a region 
> containing pending content")
>   signal(org-pending-error ("Cannot modify a region containing pending 
> content"))
>   (closure (t) (&rest _) (signal 'org-pending-error (list "Cannot modify a 
> region containing pending content")))(1 81)
>   primitive-undo(1 ((1 . 81) (t . 0)))
>   undo-more(1)
>   undo(nil)
>   funcall-interactively(undo nil)
>   command-execute(undo)

Thanks for the details.

This is what is supposed to happen: the 'undo' is trying to erase the
whole buffer content, but that buffer contains a reglock, thus,
org-pending is explicitly interrupting that operation and raising a
meaningful error instead.

What other behaviour are you expecting here ?



>>> 3. I tried M-x undo _after_ reglock was unlocked, and I got "TO REPLACE"
>>>word highlighted. I did not expect it to be highlighted.
>>
>> I couldn't get that behavior, but the undo wasn't correct either.
>>
>> org-pending is now directly instrumenting the buffer-undo-list, and
>> manually adding an undo-boundary.
>>
>> Do you still see your problem ?
>
> No, this problem is solved now.

Perfect! Thanks for testing.

(I removed the undo-boundary from org-pending; I moved it to the
example)

>>> 4. If I try to cancel the reglock, it does get canceled, but *Region
>>>Lock* buffer is not updated - Live? still displays "yes Cancel".
>>
>> It's by design.
>>
>> The function `org-pending-describe-reglock' works like
>> `describe-variable' and other similar functions.  You need to revert
>> the buffer (g) to update it.
>
> I still find it confusing.
> Mostly because it is not clear if pressing "cancel" does anything.

I added a header to make it clear that the info of the buffer is a
snapshot at a given time.  And, that the user needs to hit the usual
key 'g' to revert the buffer.

When clicking "Cancel", org-pending now aknowledges that the cancel
request has been sent, using a message.


>> The reglock is live in its buffer.
>
> What do you mean by that??
> How can reglock be active in the buffer tha

Re: Pending contents in org documents

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
> I have one suggestion though. You now do
>
> Use the function ON-OUTCOME to update the region with the outcome; if it
> is nil, set it to the function `org-pending-on-outcome-replace'.
>
> However, `org-pending' is defined via `cl-defun', so you can instead
> just put the default value for :on-outcome key and mention that it is
> the default in the docstring. Then, you do not need to do any additional
> checks in the function body; `cl-defun' will take care about assigning
> the default value.

Done.


> I tried to run your example and have several observations:
>
> 1. On failure, it is not obvious that failure happened:
>- The failure overlay disappear very quickly, and is not visible at
>   all if I happen to look elsewhere in the buffer. Maybe we can
>   simply keep it and remove the overlay on click

The current method was to 'sit-for' 0.2 seconds; the new default is to
do nothing (the overlay disappears immediately), but it's now
configurable, see below.


>- After failure, the "!" fringe indicator is visible, but it is not
>  obvious at all that user can click to get details
>  I first tried to click on the fringe itself to no avail. Then, I
>  randomly clicked on the text and got the description buffer; but
>  that was unexpected - the text I clicked did not have any
>  indication of its "clickability" - neither some kind of underline
>  face, nor an overlay or a mouse hint.

I couldn't find a way to answer a click on a fringe.  Is there a way?

About how much to decorate, it depends on the user, I guess.  For
example, when org-pending is used for org babel, it should be obvious
that one has to click on "#+RESULTS:".

The current decoration is not the best for discoverability, indeed.
But decorating the whole outcome region would be too much IMHO, and,
it might interfer with other buffer fontifications.

I've refactored the code and added two variables so that it's now
configurable, see 'org-pending-outcome-pre-display-function' and
'org-pending-outcome-post-display-function'.

I'll be happy to commit a patch that provide better defaults.

> 2. I tried to do M-x undo while the reglock was active, and got an
>error. I'd expect that undo would work, skipping the region.

I'm not sure exactly what you did, nor which error you got.  I've
noticed that the hacks (to handle indirect buffers) were flagging the
buffer as "modified" (using text properties).  I've fixed that.

Is the problem solved now ?

>
> 3. I tried M-x undo _after_ reglock was unlocked, and I got "TO REPLACE"
>word highlighted. I did not expect it to be highlighted.

I couldn't get that behavior, but the undo wasn't correct either.

org-pending is now directly instrumenting the buffer-undo-list, and
manually adding an undo-boundary.

Do you still see your problem ?


> 4. If I try to cancel the reglock, it does get canceled, but *Region
>Lock* buffer is not updated - Live? still displays "yes Cancel".

It's by design.

The function `org-pending-describe-reglock' works like
`describe-variable' and other similar functions.  You need to revert
the buffer (g) to update it.

The reglock is live in its buffer.


>>> What is the difference between "canceling" and "killing" the reglock?
>>> Do they need to be separate?
>>
>> If you cut out, from the example, the part where they differ, they do
>> look the same indeed :)
>>
>> I'm apparently failing to explain and document this correctly, as it
>> looks like a recurring topic, sorry.
>>
>> Yes, they need to be separate as they are two different operations.
>>
>>  - cancel: The *user* may request a *cancel*; it's a polite way to
>>tell org-pending that the user doesn't care anymore about the
>>outcome.  A valid implementation is to ignore the user request.
>>The default implementation is to unlock the region (sending a
>>cancel :failure using 'org-pending-send-update'): it unlocks the
>>region, ignoring why it was locked..
>>
>>  - kill: *Emacs* may have to *kill* some locks, because Emacs is
>>killed, or the lock buffer is killed.  org-pending will intercept
>>the operations of this kind, ask the user to confirm the
>>destruction, and, if confirmed, it will give a chance to the lock
>>to do some cleanup by calling the 'before-kill-function'.
>
> Does it mean that clicking "cancel" does not guarantee that the region
> will not be updated by the runnin

Re: remote calls to tables and (empty entry, NAN)

[Bruno Barbier]

Hi,

Uwe Brauer  writes:

> However if in table1 there is an empty entry in the relevant column,
> remote copies it as 0
>
> #+begin_src 
>
> #+NAME: table2
> | Name   | Ex1 | Ex2 | Ex2 | Ex4 | Ex5 | ResSh1 |
> |+-+-+-+-+-+|
> | Smith  | |   3 |   4 |   6 |   7 ||
> | Miller |   2 |  10 |   1 |   1 |   5 | 19 |
> | Wick   |   1 |   2 |   3 |  10 |   2 | 18 |
> #+TBLFM: $7=if("$2" == "nan", string(""),vsum($2..$6)); E f-2
>
>
> #+Name: total
> | Name   | ResSh1 | ResSh2 | Total |
> |+++---|
> | Smith  |  0 ||   |
> | Miller | 19 ||   |
> | Wick   | 18 ||   |
> #+TBLFM: $2=remote(table2,@@#$7)
> #+end_src
>
> Any idea how to obtain an empty entry instead of 0?

This works for me:

   #+TBLFM: $2=if(typeof(remote(table2, @@#$7)) == 12, string(""), 
remote(table2, @@#$7)); E

(everything is documented here:
   
https://orgmode.org/manual/Formula-syntax-for-Calc.html#Formula-syntax-for-Calc
)

This works too, although I'm not sure it should:

   #+TBLFM: $2=if("remote(table2, @@#$7)" = "nan", string(""), remote(table2, 
@@#$7)); E

HTH,

Bruno

Re: Using Org-mode macros as LaTeX macros

[Bruno Barbier]

Hi Sébastien,


Sébastien Gendre  writes:

> TL;DR: How can I use the Org-mode macros as LaTeX macros inside an
>export LaTeX bloc ?
>

It looks like macros are not expanded in latex export blocks, indeed.

You may define your own filter to ask Org to expand them.

With the 2 functions below, and this configuration:

 (add-to-list 'org-export-filter-export-block-functions
  'my-latex-filter-export-block)


the following org document:

 #+MACRO: orga School Name

 #+begin_export latex
 \begin{titlepage}
   Some custom LaTeX here
   This is my school: {{{orga}}}
 \end{titlepage}
 \newcommand{\orga}orga
 #+end_export

is exported like this:

 \begin{titlepage}
   Some custom LaTeX here
   This is my school: School Name
 \end{titlepage}
 \newcommand{\orga}{School Name}


Bruno. 




(cl-defun my-org-macro-expand-text (text &key templates)
  "Expand TEMPLATES in TEXT.
Assume the current-buffer is an org mode buffer.
If TEMPLATES is nil, use 'org-macro-templates'."
  (unless templates (setq templates org-macro-templates))
  (with-temp-buffer
(insert text)
(org-mode)
(goto-char (point-min))
;; Extracted from 'org-macro-replace-all'
(while (re-search-forward "{{{[-A-Za-z0-9_]" nil t)
  (let ((macro (save-excursion
 (goto-char (match-beginning 0))
 (org-element-macro-parser)))
value)
(when macro 
(let* ((value (org-macro-expand macro templates))
 (begin (org-element-begin macro)))
(delete-region
 begin
 (progn (goto-char (org-element-end macro))
  (skip-chars-backward " \t")
  (point)))
(save-excursion (insert value))
(buffer-substring (point-min) (point-max

(defun my-latex-filter-export-block (text backend info)
  "Replace macros in LaTeX export blocks."
  (when (org-export-derived-backend-p backend 'latex)
(my-org-macro-expand-text text)))

Re: Pending contents in org documents

[Bruno Barbier]

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>>>> ;;(org-pending-send-update my-rlock  (list :progress "Not ready 
>>>> yet."))
>>>> ;;(org-pending-send-update my-rlock  (list :progress "Coming 
>>>> soon."))
>>>
>>> Should the progress message always be a string?
>>
>> No.  It may currently be any data.  org-pending will format it as a
>> string that fits on one line.
>
> Please say this in the docstring of `org-pending-send-update'.

Done.



>>>> ;;(org-pending-send-update my-rlock (list :success 1))
>>>
>>> What will org-pending do with :success 1? Will it replace region with
>>> "1" or will it do something else?
>>
>> That's the job on ON-OUTCOME to convert/format/append/prepend/replace
>> some content if needed, on :success and/or on :failure.
>
> Fair. Although, it feels like a common use case to replace the region
> with :success value. Maybe the library should provide some ready-to-use
> functions that can be used as the value of :on-outcome.

I've recycled the old function used by `org-pending-user-edit',
improved it and made it the default :on-outcome handler: see
`org-pending-on-outcome-replace'.  I've simplified the example
accordingly, removing the custom :on-outcome.

I don't know any safe way to replace some text, but, I hope that will
be a good enough default.



>>> It would be nice to have an example that will also send a signal to
>>> process, as it is probably the most commonly used way to utilize
>>> org-pending.
>>
>> For my many use cases, that would always be a mistake to kill the
>> process: an OS process is always in charge of many locks.
>>
>> More importantly, to find a self-contained working readable example
>> might be a challenge.
>>
>> We could add a function 'org-pending-shell-command-on-region' in
>> org-pending, that could be used as an implementation example, like
>> `org-pending-user-edit', `org-babel-execute-src-block', etc.
>
> Yes, having pre-cooked wrappers for `org-pending' or pre-defined values
> for :on-outcome/:befire-kill-function/:user-cancel-function/etc would be 
> useful.

:on-outcome now has a better default: `org-pending-on-outcome-replace'
(see above).

The predefined values for :before-kill-function and
:user-cancel-function seem OK to me.  We will see, when using
org-pending, if some patterns need to be included in org-pending.

>From the many examples provided in the branch, do you see any that
should be included in the library as an other precooked-wrapper, that
should be included in the section "Basic use of locks" ?

I've added 'org-pending-shell-command-on-region' to my todo list.


>
> ;; ;; We lock the 'region', defining how to update it when the
> ;; ;; outcome is available.
> ;; (setq my-lock (org-pending
> ;;region
> ;;:on-outcome
> ;;(lambda (_rl outcome)
> ;;  (pcase outcome
> ;;(`(:success ,value)
> ;; ;; On :success, we replace the region with the
> ;; ;; value.
> ;; (let ((tmp-end (cdr region)))
> ;;   (goto-char tmp-end)
> ;;   (insert (format "%s\n" value))
> ;;   (setcdr region (point))
> ;;   (delete-region (car region) tmp-end)))
> ;; ...
>
> This example has a major problem if user edits the buffer text _before_
> locked region before the outcome is available.
> (car region) and (cdr region) will no longer be accurate, and your code
> will replace text in places you do not expect.
> I believe that it will be better to query region-lock object about the
> region location where we need to replace text:
>
> (setq region (org-pending-reglock-region rl))
>
> Same for reglock buffer in other examples.
>
> Then, we will keep the possibility open for org-pending to handle cases
> like killing/yanking text containing reglocks (org-refile) - org-pending
> may in future keep track of them.

I see. Good point!

But note that the region is a "read-only constant" field;
archiving/refiling live locks is forbidden.

I modified the example to rely on the reglock when possible (as
opposed to values kept from the creation time).


>
> ;; (setf (org-pending-reglock-user-cancel-function my-lock)
> ;;   (let ((this-timer my-timer))
> ;; (lambda (_rl)
> ;;   (cancel-timer this-timer)

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
> I am attaching some minor edits I'd like to propose on top of your
> latest branch.

I've applied your patch. Thanks.

> Also, some more questions.
>
>> ;;(setq my-rlock
>> ;;  (org-pending (cons (point) (mark))
>> ;;   (lambda (outcome)
>> ;; (pcase outcome
>> ;;   (`(:success ,result) (goto-char END) 
>> (insert result))
>> ;;   (`(:failure ,err) (message "Failed: %s" 
>> err))
>
> 1. It is more natural in Elisp to pass regions as two parameters: BEG
>and END, not a cons cell.

I rewrote the example so that it's complete and can be uncommented and
evaluated.

With the new example, it's less obvious which is more natural.



> 2. Note that (point) may be _after_ (mark).  AFAIU, you code assumes
>that point is always before the mark.  You may want to address this.

Right. I'm not using those in the new example: problem solved ;)



> 3. ON-OUTCOME is optional. What happens if none is provided?

No function is called. And thus, no value is returned, so no mark.
It's like calling a function that does nothing and return nil.



> 4. In the `org-pending' docstring you say "ON-OUTCOME is non-nil, call
>it with the reglock and the outcome", but the example shows a lambda
>accepting a single argument. Something is off. I'm afraid that this
>example will not work if copy-pasted.
>

You're right. Sorry. I'm now using using a fully working example.



>> ;;(org-pending-send-update my-rlock  (list :progress "Not ready 
>> yet."))
>> ;;(org-pending-send-update my-rlock  (list :progress "Coming soon."))
>
> Should the progress message always be a string?

No.  It may currently be any data.  org-pending will format it as a
string that fits on one line.


>> ;;(org-pending-send-update my-rlock (list :success 1))
>
> What will org-pending do with :success 1? Will it replace region with
> "1" or will it do something else?

That's the job on ON-OUTCOME to convert/format/append/prepend/replace
some content if needed, on :success and/or on :failure.


>> ;;(org-pending-send-update my-rlock (list :failure "Some error!"))
>
> I am slightly confused by this calling convention. Why not simply
>
> (org-pending-send-update my-rlock :failure "Some error!")

Because one message update is one value, either:
 (list :success value)
or
 (list :failure err)
or
 (list :progress data)

On :success/:failure, this message is also a valid outcome, and it's
the same value that ON-OUTCOME will receive, as one single entity.

When listing artificial calls like this, it does look suboptimal; but,
in general, it's simpler to have a real value we can store, pass
around and return.

>
>> ;;(setf (org-pending-reglock-insert-details-function my-reglock)
>> ;;  (lambda (rl _start _end)
>> ;;(insert (format "%s" (org-pending-reglock-property rl 
>> :my-prop)
>
> Are there any standard properties? It would be nice to list them in a
> table as well.

There are no standard properties.  All required properties have been
hard-coded in the cl-defstruct.

> Also, you can show an example of _setting_ :my-prop property.

I removed the use of `org-pending-reglock-property' from the example; I
think it was a mistake to introduce that advanced feature here.  The
documentation of the function `org-pending-reglock-property' explains
how to set a property.


>> ;; If the user kills a buffer, or, kills Emacs, some locks may have to
>> ;; be killed too.  The library will ask the user to confirm if an
>> ;; operation requires to kill some locks.  See the field
>> ;; `before-kill-function' of REGLOCK object, if you need to do
>> ;; something before a lock is really killed.  For example, if you like
>> ;; to kill a MY-BUFFER before MY-LOCK is killed, you can do:
>> ;;
>> ;;(setf (org-pending-reglock-before-kill-function my-reglock)
>> ;;  (lambda (_rl) (kill-buffer my-buffer)))
>
> It would be nice to have an example that will also send a signal to
> process, as it is probably the most commonly used way to utilize
> org-pending.

For my many use cases, that would always be a mistake to kill the
process: an OS process is always in charge of many locks.

More importantly, to find a self-contained working readable example
might be a challenge.

We could add a function 'org-pending-shell-command-on-region' in
org-pending, that could be used as an implementation exampl

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>> I've pushed the modification to my branch.
>
> Thanks! Let's work further on the top comment.
>
>> ;; To lock a region, you need to do something like this:
>
> I think "something like this" can be just dropped.

Sorry, I'm failing to find a simpler accurate sentence.


>> ;;1. Call the function `org-pending' with the region to lock; use
>> ;;   the ON-OUTCOME argument to tell Emacs how to update the
>> ;;   region.  Keep the returned REGLOCK (you'll need it to send
>> ;;   updates).
>
> It would be nice to provide examples of `org-pending' call right in the
> top comment. Ideally, the example should also show how to modify REGLOCK
> fields to customize its behaviour.

I added an example below the 2 steps.

>> ;;2. Start "something" that computes the new content.  That
>> ;;   "something" may be a thread, a timer, a notification, a
>> ;;   process, etc.  That "something" must eventually send a
>> ;;   :success or :failure message (using
>> ;;   `org-pending-send-update'): Emacs will update the pending
>> ;;   region (using your ON-OUTCOME) and unlock it; at this point
>> ;;   the lock is "dead" (not live-p).
>
> Please also add information about sending updates to the REGLOCK, with
> examples. Otherwise, "receiving progress" in the next section is
> surprising.

Done in the example too.

>> ... (not live-p)
>
> Please name `org-pending-reglock-live-p' - it is more clear than forcing
> the readers search it themselves.

Done.

>
>>  Interface provided to the Emacs user
>> ;;
>> ;; The library makes locks visible to the user using text properties
>> ;; and/or overlays.  It diplays and updates the status while the
>> ;; region is locked: the initial status is "scheduled", then, when
>
> It would be nice to name `org-pending-reglock-status' here and use the
> actual status values - :scheduled, :pending, :success, :failure.
> Ideally, in table/list explaining what happens with buffer text,
> overlays, and user interaction, when REGLOCK has each of the listed
> status values.

I added a table describing the possible status values, the valid
updates, etc in the section above.


>
>> ;; receiving progress it becomes "pending" (with progress information
>> ;; if any).  Emacs allows to diplay a description of the lock.  From
>> ;; that description, the user may request to cancel that lock; see the
>> ;; field `user-cancel-function' of the REGLOCK object if you need to
>> ;; customize what to do on cancel.
>
> It is not very clear how user can interact with "description".
> Is it a  tooltip? A window? Something else? Please give a bit more details.

I added that it's like the function `describe-package'.

> Also, when "cancel" is requested, it is a good idea to state that
> `user-cancel-function' is called and describe what it does by default.

The function `user-cancel-function' is not always called; Emacs will
only call it if the lock is still live (see `org-pending-cancel').

I added a short sentence describing what is done by default (the
documentation of the field `user-cancel-function' of the reglock
object already fully document `user-cancel-function').


>> ;; When receiving the outcome (success or failure), after unlocking
>> ;; the region, the library may leave information about the outcome
>
> "may"?? Or does it always leave the information (by default)?

To mark the outcome, org-pending needs to know where it is.  If the
ON-OUTCOME function (see `org-pending` documentation) returns the
outcome region, org-pending will add a mark, else not.  By default,
ON-OUTCOME is nil, meaning no outcome region and no outcome marks.  I
added a sentence about that.


>> ;; (using text properties/overlays).  If that outcome information is
>> ;; (still) displayed, Emacs allows to display a description of that
>> ;; lock.  From that description, the user may decide to "forget" that
>> ;; lock; "forgetting the lock" removes the outcome visual marks, and,
>
> Is "forgetting" customizeable like `user-cancel-function'?

No.

>> ;; it allows Emacs to discard any information related to this lock.
>
> What does it mean?

Emacs is free to delete any data related to this lock.  This lock will
not be available anywhere anymore for the user. Concretely, this lock
is remove from the list of known locks, and, eventually, Emacs should
hopefully garbage collect the related data.

>
>> ;; The descri

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Thanks for the review Ihor!

>> Thanks. I've improved the documentation of `org-pending' to mention
>> that one may want to customize the following fields of a reglock:
>> before-kill-function, user-cancel-function and
>> insert-details-function.  And, also, I added that one can attach
>> custom properties using the "properties" field.
>
> Thanks, but I still feel confused.
> May you:
>
> 1. Explain what does "kill" and "cancel" mean in the context of the
>REGLOCK.

I tried to document that as part of the function `org-pending. That is:

   |- Emacs may have to kill your locks; see the field
   |  `before-kill-function' if you wish to do something before your
   |  lock is killed.

In other words, Emacs has to kill the locks, because the user decided
to kill Emacs or some buffers.  The user has no say about this.  This
is when org-pending calls the private function
`org-pending--forced-kill'.
   
   
   |- The user may ask Emacs to cancel your lock; see the field
   |  `user-cancel-function' to override the default cancel function.

The user is not interested anymore by the lock.  He makes a polite
request to the library that locked the region to unlock it, the sooner
the better.

I added some documentation about them to the top level commentary.
   
> 2. Add information about visual hints, "kill", and "cancel" to the
>top-level commentary.
>
> For now, when reading the top commentary:
>
> ;; This library contains an API to lock a region while it is "being
> ;; updated"; the content of the region is "pending" and cannot be
> ;; modified.  It will be updated, later, when the new content is
> ;; available.
>
> I have an impression that the only side effect of "locking" is that the
> region becomes read-only. It is not clear at all that any other
> visual indication will be provided.

I added a description of the user interface at top-level.


[...]
> Yes, I do think that top commentary should explain this.
> The very idea that lock may be "canceled" or "killed" is important when
> designing Elisp code that uses the org-pending library.
>

Both "killed" and "canceled" are now described in the user interface
top-level section.


[...]
>
> I hope that the above clarifies what I am looking for.

I think it did.  And I even hope the improved top-level documentation
shows it :)


I've pushed the modification to my branch.

Thanks again for the review, your time and useful comments,


Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi Ihor,

Thanks for the review.

I've pushed a new version, hoping to decrease the number of dislikes
;-)

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>>> I have a further request on interaction with penreg objects.
>>> I feel that it is not ideal that overlays associated with penreg objects
>>> cannot be fully controlled by the callers.
>>
>> I'm trying to limit the public API surface.  I don't think we should
>> leak that we are currently using a mix of overlays and text
>> properties.
>
> Let me rephrase my concern - I do not like that after reglock is no
> longer live (got success/failure signal), there is no way to clean up the
> visual hints associated with this particular reglock.
[]

For the org-pending library, "live" means "locked".  Once the outcome
is known, it can't be "live" anymore (it's unlocked); as it's not
reusable, it's "dead".

As the region is not locked anymore, the lock properties/fields can't
be trusted anymore.

But see below about removing the visual outcome hints of a given
reglock.

>>> 2. Act on the outcome overlays - there is currently no way to remove
>>>them using penreg object.
>>
>> I've added a funcion `org-pending-delete-outcome-marks' to manually
>> delete outcome marks that are in a given region.
>>
>> Else, everything is handled automatically. Once the outcome is known,
>> the reglock is dead (not live-p).  org-pending may leave outcome marks
>> about the outcomes (outcome marks are optional).  The outcome marks
>> automatically disappear if the user remove the section, or, if a new
>> lock is created for the same region.
>
> I do not like this.
> I'd like the Elisp program that creates the reglock to be able to
> clean up any visual hints associated with it. > A function doing it for a
> given region cannot do this AFAIU.

ok. I've added the function `org-pending-reglock-delete-outcome-marks,
that will delete the outcome visual hints for a given reglock, if
there are some.

I updated how the lock is described to the user
(org-pending-describe-reglock): I added a button "Forget" (if the lock
is dead, that removes the outcome marks), and I added a "Cancel"
button if the lock is still live.


>>>Maybe :cancel signal? Canceled penreg
>>>objects can then be garbage-collected from the manager.
>>
>> Cancel is handled by sending a failure message (see
>>  `org-pending-cancel').  It's customizable using the reglock field
>>  ~org-pending-reglock-user-cancel-function~, which can decide what to
>>  do (like kill a process) and which can send a better outcome.
>>  Standard 'cancel' leaves a failure outcome mark.
>
> Note that this function is not documented anywhere other than in reglock
> class documentation.

Thanks. I've improved the documentation of `org-pending' to mention
that one may want to customize the following fields of a reglock:
before-kill-function, user-cancel-function and
insert-details-function.  And, also, I added that one can attach
custom properties using the "properties" field.

> In general, I am confused about your overall design
> of the user interaction with the locks.
> The updated top commentary explains well how Elisp programs can send
> data to the locks, but it does not say anything about how Elisp programs
> can receive the data.

An elisp program, that uses org-pending, must update the locks using
`org-pending-send-update'.  That program does not receive any data
from the lock; it may customize Emacs behavior using the reglock
fields mentioned above: before-kill-function, user-cancel-function and
insert-details-function.

Hopefully, it's clearer now with the improved documentation of the
org-pending function.

Just let me know if you still think that the top commentary should
explain this.  Thanks.


> Also, I'd like to see more information in the top commentary about what
> are the "visual hints" displayed to the user and how to configure them.

If you think the current "visual hints" are good enough and could be
shipped as-is, in a first version (indirect buffers, etc.); I could
work on documenting them better.  What kind of configuration are you
thinking about ? just the faces ? or more advanced configurations ?


[...]

>>> Is there any reason why you hide the extra information behind :-alist
>>> filed? Why not directly adding extra fields with proper documentation?
>>
>> To hide them, indeed :)
>
>> The API for 'get-status and 'get-live-p are
>> `org-pending-reglock-status' and `org-pending-reglock-live-p' (they
>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Rudolf Adamkovič  writes:

>
> +1 for the full name.
>
> Searching 'M-x' for 'region' gives 229 results on my Emacs, so there is
> a precedent.  In fact, when I first read the name 'reglock', I took
> 'reg' for *not* a region, but register or registry, precisely because
> Emacs consistently spells the word out in full.
>

Thanks for your input.

Note that "reglock" is the current technical name, an implementation
detail, and for developer only.  The only current command using that
term (that shows up using M-x) is a debugging internal command
(org-pending--describe-reglock-at-point).

I'm thinking about using just "lock" but I'll keep using "reglock" for
now, as it's easier to search/replace, until the public API is
finalized.

Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>> I've pushed an update that should address most of your comments.
>
> Thanks!
>
>> I've found a better name: I'm now calling it a "lock".  So I renamed
>> "PENREG" into "REGLOCK" as "region lock".  The structure is now
>> `org-pending-reglock'.
>
> I slightly dislike short names and would prefer region-lock, but not a
> big deal - it is just my personal style preference.

It's more about being one word than being short, and being a new and
opaque word.  REGLOCK is a new technical term; it's definition is the
cl-defstruct.  For normal use, this is just a black box to pass
around.

I'll probably switch to using just "lock".



>> I've renamed org-pending-ti-send-update to org-pending-send-update
>> (now that the task control is gone, the prefix becomes useless).
>
> I have a further request on interaction with penreg objects.
> I feel that it is not ideal that overlays associated with penreg objects
> cannot be fully controlled by the callers.

I'm trying to limit the public API surface.  I don't think we should
leak that we are currently using a mix of overlays and text
properties.


> In particular, I'd like to see some way to
>
> 1. Create penreg object without locking the region, so that scheduled-at
>time is not set immediately and status overlay is not displayed.
>Then, `org-pending-send-update' could send :schedule signal to
>perform actual lock.

Using the term "region" was confusing, sorry.  That's why I switched
to region "lock".  I don't think there is a use to create a lock that
doesn't lock.

Also, that might be tricky to implement: `org-pending-send-update' is
called asynchronously, from the user point of view.  Having regions
that suddenly become locked, independently of what the user is
currently doing (if we implement the :schedule message), might be
difficult.

What use do you have in mind ?


> 2. Act on the outcome overlays - there is currently no way to remove
>them using penreg object.

I've added a funcion `org-pending-delete-outcome-marks' to manually
delete outcome marks that are in a given region.

Else, everything is handled automatically. Once the outcome is known,
the reglock is dead (not live-p).  org-pending may leave outcome marks
about the outcomes (outcome marks are optional).  The outcome marks
automatically disappear if the user remove the section, or, if a new
lock is created for the same region.


>Maybe :cancel signal? Canceled penreg
>objects can then be garbage-collected from the manager.

Cancel is handled by sending a failure message (see
 `org-pending-cancel').  It's customizable using the reglock field
 ~org-pending-reglock-user-cancel-function~, which can decide what to
 do (like kill a process) and which can send a better outcome.
 Standard 'cancel' leaves a failure outcome mark.
 
I've added garbage collections of useless reglocks (success or
failure): see `org-pending--mgr-garbage-collect'.



> Also, the top-level commentary is getting incomplete and out-of-sync at
> this point. May you work towards more detailed top-level description of
> the library?

Sorry. I tried hard to keep it in sync with all the modifications.

I found it too much work, and, possibly overwhelming for the reader,
to explain everything in the top-level "Commentary" section.

That's why I deleted everything that wasn't mandatory to understand
the core features.

Everything should be documented as elisp documentation strings,
following the documentation of `org-pending' and
`org-pending-send-update', and, from code comments.


> May you work towards more detailed top-level description of
> the library?
> This will make your ideas more explicit and make life
> easier for me during the further review (now, I have to guess what you
> meant by some parts of the code).

Sorry, and thank you again for your time.

I tried to improve the overall documentation.  I hope it's going to be
easier for you, and others.


>>> HANLDER will be another object we may expose via something like
>>> (org-pending-handler (&key on-success-function on-cancel-function on-await 
>>> on-insert-logs) ...)
>>> Then, PENREG will call appropriate handler function as needed.
>>
>> As the task-control is now gone:
>>   - get/await is gone,
>>   - cancel is now a hook/function of REGLOCK,
>>   - insert-details is now a hook/function too of REGLOCK.
>>> ...
>>> Yes. Lexical context is implicit and harder to debug, while storing
>>> necessary data explicitly in the struct slots is more robust as we are
>&

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi Ihor,

Thanks for your review and detailed explanations.

I've pushed an update that should address most of your comments.

Let me answer point by point below.

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>>> I feel that org-pending-penreg (org-pending-) is
>>> redundant. Maybe better use org-pending-region?
>>
>> PENREG is the name of the structure; the "org-pending" is the
>> mandatory library prefix; this mechanically gives the name.  A PENDREG
>> object is not a "region" in Emacs sense.
>>
>> Do you see a better name for the structure PENREG, so that it doesn't
>> sound like a "region" ?
>
> Library prefix is also a part of the name and delivers useful
> information. "org-pending-region" and "region" and not the same names.
>
> We make use of prefix semantic in various places:
> - org-export-backend, implying not just "backend", but also "export"
> - org-cite-processor, implying not just "processor", but also "cite"
> - org-lint-checker - "org-lint" + "checker"
> - org-element-deferred - "org-element" + "deferred"
>
> So, there is no need to duplicate information from the prefix - it is an
> integral part of the struct name. Doing otherwise would go again the
> existing naming in Org code base.

Got it. Thanks for the explanation.

I've found a better name: I'm now calling it a "lock".  So I renamed
"PENREG" into "REGLOCK" as "region lock".  The structure is now
`org-pending-reglock'.


>>> 1. It is not clear why you need a separate ~virtual~ field. When
>>>~region~ is nil it already implies that the pending region is
>>>virtual.
>>
[...]
> Also, ~virtual~ field is unused. So, maybe we can even drop it
> completely. We can always add new fields in future, if a need arises.

The region is not allowed to be nil anymore. All "virtual" issues
are solved! :)


>>> 3. ~source~ field must match the ~region~ marker buffer. Then, why do we
>>>need this field at all? May as well just use (marker-buffer (car region))
>>
>> The "source" is the region requesting the update.
>
> The docstring of `org-pending' states that it is a buffer position:
>
> The SOURCE is the buffer position that requested this pending region.

Sorry. It was, yes.

>> ... The pending region
>> is the "target" of the update, i.e. the region that will be updated.
>>
>>
>> For example, in DEMO_ONLY, with org-babel, these 2 regions are never
>> the same:
>>1. the source is the source code block,
>>2. the target (pending region) is the result region.
>
> I am wondering why source must be a buffer position.
> What if we want to mark a region pending for some task not associated
> with a source? And why do we need to know the source at all?

Good point. It was to allow the user to quickly jump to the source
code block; I removed it from org-pending.



>>  2. insert-details: If, and only if, the user decides to
>>  investigate what happened, Emacs will ask the task if it has any
>>  details to add, that might help the user (like exit-code for an
>>  OS process, stderr for an OS process or link to a log file, etc.)
>
> I have to say that I am confused about "insert-details" part. Mostly
> because it is not per se connected to the associated task. It is rather
> an additional handler used to provide debug information about the task
> status and outcome.
> AFAIU, it is conceptually very similar to HANDLE-RESULT function.

You're right: it was confusing.  It's now like a hook, that
org-pending-describe-reglock will use.  It should now be clearer that
it's for "information purposes" only.


> I think that rather than handing HANDLE-RESULT and also TASK-CONTROL, we
> may reduce everything to a single "handler" object that will serve as a
> way for PENREG to communicate back to Elisp. That way, we do not need to
> have a concept of a "task".

I removed the task-control, and, the concept of "task".  HANDLE-RESULT
is gone from org-pending.  `org-pending' has a new keyword: :on-outcome
that will allow to do anything, both on success and on failure.

> Instead, it will be a familiar async API
> with ability to (1) create (2) send signals to (3) receive signals from
> PENREG object.
> `org-pending' will be the entry point to create PENREG object.
>
> `org-pending-ti-send-update' (or maybe simply
> `org-pending-send-update'?) will be a way to send data to PENREG object.
>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Ihor Radchenko  writes:

Thanks for your review Ihor!

> Bruno Barbier  writes:
>
>> I rewrote the API, rename many things, moved the code around and
>> sorted everything into heading/subheading sections.  This is hopefully
>> less confusing and a lot simpler; and the documentation hopefully
>> explains better how to use it.
>
> Thanks! It does look more clear.
>
>> The updated section "Commentary", in org-pending, describes the 3 steps
>> that are needed to mark and use a pending region: a PENREG (I've renamed
>> "PINFO" to "PENREG", for PENding REGion, more specific).
>
> I feel that org-pending-penreg (org-pending-) is
> redundant. Maybe better use org-pending-region?

PENREG is the name of the structure; the "org-pending" is the
mandatory library prefix; this mechanically gives the name.  A PENDREG
object is not a "region" in Emacs sense.

Do you see a better name for the structure PENREG, so that it doesn't
sound like a "region" ?


>> WDYT of this version ?
>
> Comments on org-pending-pendreg struct:
>
> 1. It is not clear why you need a separate ~virtual~ field. When
>~region~ is nil it already implies that the pending region is
>virtual.

It's a constant.  Calling a function looks more like we need to
recompute it each time, and, we could even change it.  And
cl-defstruct writes the function for us.

Do you prefer a manually written function ?


> 2. ~id~ field is semi-internal and assumed to be a number.
>Maybe we can do something more similar to Emacs process API:
>
>(make-process &rest ARGS)
>...
>:name NAME -- NAME is name for process.  It is modified if necessary
>to make it unique.
>
>We can replace id with human-readable name that can also be supplied
>when creating a pending region

Good idea. Done.


> 3. ~source~ field must match the ~region~ marker buffer. Then, why do we
>need this field at all? May as well just use (marker-buffer (car region))

The "source" is the region requesting the update.  The pending region
is the "target" of the update, i.e. the region that will be updated.


For example, in DEMO_ONLY, with org-babel, these 2 regions are never
the same:
   1. the source is the source code block,
   2. the target (pending region) is the result region.

I tried to improve the documentation of `org-pending-penreg' (source &
region).


> On the design of ~org-pending~ and ~org-pending-task-connect~:
>
> 1. I feel confused about the overall design of the interaction between
>pending region and the associated task.
>
>Functions like ~org-pending-task-send-update~ imply that pending
>region is rather decoupled from from associated task and the task
>should arrange manually for sending updates to the pending region
>object.

Exactly: the task implementation must use these ~org-pending-task-XXX~
functions to send updates to one (or more) pending region(s).


>On the other hand, there is ~task-connection~ that is used to kill
>associated task/process or to "await" for it. This time, pending
>region is strongly coupled with the task, killing it upon deleting
>the pending region.


These are optional features; and only ~org-pending~ will know if and
when those might be useful.  That's why the task needs to provide
callbacks here.
 1. cancel: Emacs may, in exceptional cases only,
 send a "cancel" to the task, meaning, "The user destroyed the
 pending region, and thus, Emacs will not use any update for it".

 2. insert-details: If, and only if, the user decides to
 investigate what happened, Emacs will ask the task if it has any
 details to add, that might help the user (like exit-code for an
 OS process, stderr for an OS process or link to a log file, etc.)

 3. get (await): It's an (unofficial) way, (in the degenerate case
 where the task implementation gives up on asynchronicity) to
 block until the outcome is available.  `org-pending' itself
 doesn't use it; DEMO_ONLY uses it with org-babel to define the
 synchronous executions.

>I think that we need more (optional) coupling between pending region
>and the associated task. We should be able to get more information
>about the task from pending region side - get logs, current status,
>exit status, etc.


>More specifically, I think that we need (1) allow to pass task as an
>argument for ~org-pending~.

That's actually what I started with, but, I couldn't make it work.

Breaking it like this is what allowed me to get the most generic and
simplest API that works for anything: threads, callbacks, OS processes,
timers.

If org-pending tak

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>> Would "lisp/org-pending.el" be OK ?  Or do you see a better place/name ?
>
> org-pending sounds fine. Maybe org-pending-text to be more explicit.

I've picked: "org-pending"; it's shorter and can by used as a prefix.
Thanks.


> Similar to process API, your library provides means to interact with
> process associated with the pending text - create them (40.4 Creating an
> Asynchronous Process), handle feedback (40.9.2 Process Filter
> Functions), handle on-done/fail/etc (40.10 Sentinels: Detecting Process
> Status Changes), kill/cancel execution (40.5 Deleting Processes, 40.8
> Sending Signals to Processes), insert-log (strerr in Emacs processes)
> list pending (40.6 Process Information), org-pending-on-kill-buffer
> (40.11 Querying Before Exit), get information (`process-attributes',
> 40.12 Accessing Other Processes)
>
> I simply went across 40 Processes section of Elisp manual and I see
> parallels for pretty much every subsection; but the terminology _and
> API_ are different.
>

I rewrote the API, rename many things, moved the code around and
sorted everything into heading/subheading sections.  This is hopefully
less confusing and a lot simpler; and the documentation hopefully
explains better how to use it.

The updated section "Commentary", in org-pending, describes the 3 steps
that are needed to mark and use a pending region: a PENREG (I've renamed
"PINFO" to "PENREG", for PENding REGion, more specific).

I tried to stick with the process terminology when relevant (mostly
kill-query and sentinel).

It doesn't really match the process API, but, now, it should be easier to
understand why.


>> I'm OK to rewrite it using a cl-defstruct (to be honnest, I just
>> flipped a coin to decide between a struct or an alist :) ).
>
> I think that I need to elaborate.
> It is not necessarily a matter of alist vs. struct, but more about the
> API. I do not like the idea of mutating the alist entries directly. I'd
> rather expose an API to work with "pending" as an opaque object - send
> signals to it, get status, etc.
>
> struct is slightly better here because it automatically defines setters
> and getters for all the slots without a need to write extra boilerplate
> code.

Done. Thanks!

I added a function `org-pending-user-edit': it allows to edit a region
using org-pending and `string-edit' (see org-pending.el).

I also added an other execution engine (DEMO_ONLY), that allows to
execute any elisp asynchronously, using callbacks (see
`my-use-callbacks-schedule', in my-async-tests.el), and in
my-async-tests.org, near the multithreading examples.

To test, follow the org file [my-async-tests.org] (see direct link below).

WDYT of this version ?

Thanks!

Bruno

[I've forced push the new version, squashing everything into simple
commits, if you need the old version, just ask and I'll push a copy
there.]


[my-async-tests.org]:
 
https://framagit.org/brubar/org-mode-mirror/-/blob/bba-pending-contents/scratch/bba-pending-contents/my-async-tests.org?ref_type=heads

[repo & branch]:
 
https://framagit.org/brubar/org-mode-mirror/-/tree/bba-pending-contents?ref_type=heads
 bba-pending-contents

Re: [BUG] org-babel-execute-subtree removes all overlays (org-fold-save-outline-visibility) [9.7-pre, gfb61e8]

[Bruno Barbier]

Ihor Radchenko  writes:

>
> Thanks for reporting!
> Fixed, on bugfix and main.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=8b73c8b98
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=712ef988c

It now works.

Thanks Ihor!


> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at .
> Support Org development at ,
> or support my work at 

[BUG] org-babel-execute-subtree removes all overlays (org-fold-save-outline-visibility) [9.7-pre, gfb61e8]

[Bruno Barbier]

Hi,

Org removes all overlays when calling org-babel-execute-subtree.

Here is a way to reproduce:

#+begin_src elisp
  (with-temp-buffer
(let (ovl
  (statuses nil)
  (live-p (lambda (xo) (or (and (overlay-buffer xo) "alive") 
"dead" 
  (insert "* Mandatory heading")
  (org-mode)
  (setq ovl (make-overlay (point-min) (point-max)))
  (push (list "before"  (funcall live-p ovl))  statuses)
  (org-babel-execute-subtree)
  (push (list "after"  (funcall live-p ovl)) statuses)
  (nreverse statuses)))
#+end_src

#+RESULTS:
| before | alive |
| after  | dead  |

Org should not remove overlays it doesn't own.

After investigation, it seems that any use of
`org-save-outline-visibility' will call:

(org-fold-core-region START END nil)
^^^

which will, indeed, remove any overlay (except for those that are
both 'org-invisible and 'invisible).

Thanks,

Bruno


Emacs  : GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo 
version 1.18.0)
 of 2024-03-12
Package: Org mode version 9.7-pre (release_9.6.20-1281-gfb61e8.dirty @ 
/home/bruno/work/open-src/emacs/orgmode/orgmode/lisp/)

current state:
==
(setq
 org-yank-image-file-name-function 'org-yank-image-autogen-filename
 org-persist-before-write-hook '(org-element--cache-persist-before-write)
 org-html-format-headline-function 'org-html-format-headline-default-function
 org-html-format-drawer-function #[514 "\207" [] 3 
("/home/bruno/work/open-src/emacs/orgmode/orgmode/lisp/ox-html.elc" . 24162)]
 org-latex-format-inlinetask-function 
'org-latex-format-inlinetask-default-function
 org-speed-command-hook '(org-speed-command-activate 
org-babel-speed-command-activate)
 org-persist-after-read-hook '(org-element--cache-persist-after-read)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-latex-format-headline-function 'org-latex-format-headline-default-function
 org-latex-format-drawer-function #[514 "\207" [] 3 
("/home/bruno/work/open-src/emacs/orgmode/orgmode/lisp/ox-latex.elc" . 37582)]
 org-mode-hook '(#[0 "\300\301\302\303\304$\207" [add-hook 
change-major-mode-hook org-fold-show-all append local] 5]
 #[0 "\300\301\302\303\304$\207" [add-hook 
change-major-mode-hook org-babel-show-result-all append local] 5]
 org-babel-result-hide-spec org-babel-hide-all-hashes)
 org-confirm-shell-link-function 'yes-or-no-p
 org-odt-format-headline-function 'org-odt-format-headline-default-function
 org-archive-hook '(org-attach-archive-delete-maybe)
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-occur-hook '(org-first-headline-recenter)
 org-html-format-inlinetask-function 
'org-html-format-inlinetask-default-function
 org-ascii-format-drawer-function #[NON PRINTABLE]
 org-bibtex-headline-format-function 'org-bibtex-headline-format-default
 org-link-parameters '(("attachment" :follow org-attach-follow :complete 
org-attach-complete-link)
   ("id" :follow org-id-open :store org-id-store-link-maybe)
   ("eww" :follow org-eww-open :store org-eww-store-link)
   ("rmail" :follow org-rmail-open :store 
org-rmail-store-link)
   ("mhe" :follow org-mhe-open :store org-mhe-store-link)
   ("irc" :follow org-irc-visit :store org-irc-store-link 
:export org-irc-export)
   ("info" :follow org-info-open :export org-info-export 
:store org-info-store-link :insert-description
org-info-description-as-command)
   ("gnus" :follow org-gnus-open :store org-gnus-store-link)
   ("docview" :follow org-docview-open :export 
org-docview-export :store org-docview-store-link)
   ("bibtex" :follow org-bibtex-open :store 
org-bibtex-store-link)
   ("bbdb" :follow org-bbdb-open :export org-bbdb-export 
:complete org-bbdb-complete-link :store
org-bbdb-store-link)
   ("w3m" :store org-w3m-store-link) ("doi" :follow 
org-link-doi-open :export org-link-doi-export)
   ("file+sys") ("file+emacs") ("shell" :follow 
org-link--open-shell)
   ("news" :follow
#[ ---NON PRINTABLE--- ]
  
("/home/bruno/work/open-src/emacs/orgmode/orgmode/lisp/ol.elc" . 52731)]
)
   ("mailto" :follow
#[ ---NON PRINTABLE--- ]
)
   ("https" :follow
#[ ---NON PRINTABLE--- ]
)
   ("http" :follow
#[ ---NON PRINTABLE--- ]
)
   ("ftp" :follow
#[ ---NON PRINTABLE--- ]
)
   (

Re: ob-haskell and changing executables?

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>>> If you've gotten this far you probably know more
>>> about the Haskell Babel situation than you ever wanted to, but maybe you
>>> can sniff out where this hardwire is happening.
>>
>> It's not hard coded (there is quite a lot of code just to guess the
>> right interpreter).  ob-haskell delegates the task to the
>> haskell-mode package; that's where you should be looking for to
>> configure it.
>
> Only for sessions. For src blocks with :session none, we directly call
> `org-babel-haskell-compiler'.

Good point.  I didn't check that execution path (probably because I
assumed the OP was speaking only about GHCi).

But, from what I see (function `org-babel-execute:haskell'), for
Haskell, we switch to the compiler only if explicitly requested
(':compile' option is "yes").  If not, we use the interpreter for both
with and without sessions.

Am I missing something ?

Thanks,

Bruno


>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: [Question] Learning to use Org Element API setters and how to ignore properties

[Bruno Barbier]

Hi Antonio,

Antonio Romano  writes:

[...]
> The :todo-keyword contains data about face and properties which are
> only relevant for viewing the document in the buffer and not for the
> .org file content itself. How can I read its text without any property
> info attached to it?

You could remove them using the function `substring-no-properties'.

Bruno

Re: How to get in-buffer setting from Elisp code

[Bruno Barbier]

Hi Sébastien,

Sébastien Gendre  writes:

> Hello,
>
> How can I access to an in-buffer setting value, from Elisp code ?
>

IIUC, they are called "Buffer-Local Variables".

Using 'buffer-local-value' should give you the value of a variable in a
buffer.

See the manual for more:
  (info "(elisp) Buffer-Local Variables")


Bruno

Re: ob-haskell and changing executables?

[Bruno Barbier]

Hi Laurence,

Laurence von Bottorff  writes:

> I'm on Debian 12 and I just started using Haskell's ghcup tools, leaving
> the stack tools behind, as advised these days. ghcup puts executables for
> Haskell such as ghc, ghci (REPL), cabal, etc. in its ~/.ghcup/bin
> directory. Next, to stop using the stack tools that have executables in
> /usr/bin/ you must change your PATH to go to ~/.ghcup/bin first. But when I
> try a Babel code block, ob-haskell seems to have the /usr/bin versions
> hardwired somewhere and calls up the old ghci REPL. Searching through Emacs
> Customize Haskell was confusing and my init only had one relevant entry
> anyway, which didn't help when I changed it.
>
[...]

> If you've gotten this far you probably know more
> about the Haskell Babel situation than you ever wanted to, but maybe you
> can sniff out where this hardwire is happening.

It's not hard coded (there is quite a lot of code just to guess the
right interpreter).  ob-haskell delegates the task to the
haskell-mode package; that's where you should be looking for to
configure it.

Also, if your Org file is not inside your haskell project, using the
:dir option may help haskell-mode to guess it right.

Else, did you try to customize haskell-process-type ?

There is also the option 'haskell-process-path-ghci', so, you should be
able to set the path manually too.  Or you've already forced it to
"/usr/bin/ghci" ?

Bruno


> -- 
> ⨽
> Lawrence Bottorff
> Grand Marais, MN, USA
> borg...@gmail.com

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

> Thanks!
> I have some minor concerns about implementation, but you clearly
> demonstrated the things can be working in general.

Thanks!

> While reading the library header and `org-pending' docstring (btw, it
> should probably be a separate library, not a part of org-macs),

I was feeling more and more like squatting the wrong file :-)

Would "lisp/org-pending.el" be OK ?  Or do you see a better place/name ?


> I felt confused about terminology and also had déjà vu's about what
> org-pending does.

> More or less, org-pending implements Elisp asynchronous process control,
> but the processes are associated with region, not the whole buffer.
> Hence, I feel that we should adopt terminology similar to the existing
> terminology for processes - process filters, process sentinels, sending
> signals to processes, etc. And similar API.

The current `org-pending' doesn't know what a process is, or a thread.
The job of org-pending is only to let Emacs know that a given region
will be updated at some point later (much like what
`org-edit-src-code' does, when a source block is being edited
manually).

I've re-read the Elisp manual about asynchronous processes.  The only
thing that seemed obvious to me, was to replace "feedbacks-handler"
with "sentinel".  I also tried to simplify things and use REGION
instead of CONTENT where it make sense.  Thanks.

Do you see something else ?



> Also, I am not sure if I like the idea of exposing raw PINFO alist and
> mutating it. In particular, I have doubts about mutating CONTENT.
> What we might do instead is implement PINFO as struct with custom
> accessors/setters.

I went for this simple alist because it's more flexible (easy to add new
bindings) and less code than using a struct.  And Org already uses lists
like this (params, info, etc.).  Mutating CONTENT (or other essential
data) would be a very bad idea indeed; but it's like that for pretty
much everything with elisp.

I'm OK to rewrite it using a cl-defstruct (to be honnest, I just
flipped a coin to decide between a struct or an alist :) ).

I'll push my new version after relocating everything into its own
file. Let me know if lisp/org-pending.el is OK.

Thanks.

Bruno

Re: Things got very slow: profiler output

[Bruno Barbier]

William Denton  writes:

> On Thursday, February 29th, 2024 at 04:25, Bruno Barbier 
>  wrote:
>
[...]
> I checked:
>
> |  Its value is (latex)
> |  Original value was nil
>
> I didn't have a chance to dig back into this but now I see other people are 
> reporting slowness and I hope this helps.
>
[...]

Is it slow too when using the default config (emacs -Q and no customization)
(see https://orgmode.org/org.html#Feedback) ?


Bruno


> I'm happy to try anything else ...
>
> Bill
>
> --
> William Denton
> https://www.miskatonic.org/
> Librarian, artist and licensed private investigator.
> Toronto, Canada

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi,

Bruno Barbier  writes:

> Ihor Radchenko  writes:
>
>> Bruno Barbier  writes:
>>
>>>> Overlays are not transferred when a new indirect buffer is created (for
>>>> example, by org-capture, or by user). So, it will be (1) impossible to
>>>> see pending overlays in indirect buffers; (2) user edits of pending text
>>>> from indirect buffer will not be prevented.
>>>
[...]
> I now think that overlays are the right way; the /pending content/ is
> attached to one buffer: a base or a clone; this is for the user to
> decide.
>
> I will manually add text properties, below the overlay, to mark the text
> as /pending/, so that pending contents will be visible and read-only in
> all other buffers, base or indirect ones.
>
> Cloning buffers is easy to test. I'm not sure which scenario I should
> use to test org-capture though.
>
> I'll update my branch with that improvement soon.
>
>
> Thanks Ihor!
>
> Bruno



Hi,

After some work, some bug fixes and a few segfaults (using indirect
buffers, see bug#69529), I pushed a new version.  The main changes are:

• Handle indirect buffers: the pending content belongs to the buffer
  that started it (using text properties to mirror overlays).
• Use the fringe to indicate success or failure.
• Describe the pending content (past&present) when the user clicks it
  (pending, success or failure, time, duration, log, etc.).
• Improve the logging API and provide examples (may be used to collect
  stderr for example).

As before, the org file describes how to test it, see the file
[scratch/bba-pending-contents/my-async-tests.org] (direct link below).

Comments, critiques, ideas, corrections are most welcome.

Thanks,

Bruno


[scratch/bba-pending-contents/my-async-tests.org]
<https://framagit.org/brubar/org-mode-mirror/-/tree/bba-pending-contents/scratch/bba-pending-contents/my-async-tests.org>

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>>> Overlays are not transferred when a new indirect buffer is created (for
>>> example, by org-capture, or by user). So, it will be (1) impossible to
>>> see pending overlays in indirect buffers; (2) user edits of pending text
>>> from indirect buffer will not be prevented.
>>
>> If I understand correctly, you would prefer a solution that relies on
>> text properties ? (I didn't leak the overlay details in the API, so, it
>> should not be too hard to switch to text properties, except for possible
>> conflicts with other existing properties).
>
> Yes, I think. Might also try to mirror overlays, but that's problematic
> in practice.

I now think that overlays are the right way; the /pending content/ is
attached to one buffer: a base or a clone; this is for the user to
decide.

I will manually add text properties, below the overlay, to mark the text
as /pending/, so that pending contents will be visible and read-only in
all other buffers, base or indirect ones.

Cloning buffers is easy to test. I'm not sure which scenario I should
use to test org-capture though.

I'll update my branch with that improvement soon.


Thanks Ihor!

Bruno

Re: Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi Ihor,


Ihor Radchenko  writes:

[...]
>> I've tried to fully describe the feature in the new section "Pending
>> contents", in the file `lisp/org-macs.el'.
>
> I have one general concern about the implementation.
>
> Overlays are not transferred when a new indirect buffer is created (for
> example, by org-capture, or by user). So, it will be (1) impossible to
> see pending overlays in indirect buffers; (2) user edits of pending text
> from indirect buffer will not be prevented.

If I understand correctly, you would prefer a solution that relies on
text properties ? (I didn't leak the overlay details in the API, so, it
should not be too hard to switch to text properties, except for possible
conflicts with other existing properties).

What about markers ? Are they OK ?

Thanks,


Bruno

Re: Things got very slow: profiler output

[Bruno Barbier]

Bruno Barbier  writes:

> Hi William,
>
> William Denton  writes:
>
>> I rebuilt Org and Emacs from the development trees and something is wrong, 
>> because some Org files I use regularly have become incredibly slow to use.  
>> I rarely use the profiler and don't know what to make of what it says, but I 
>> opened a file and ran it while I moved around and expanded and collapsed 
>> some headings for a minute or so.  (It was so slow that it took me a minute 
>> to do what would usually take me a few seconds.)  I'll paste the results 
>> below.  Does that say anything useful?  There is a little LaTeX in the file 
>> but not much.
>>
>> Any help on interpreting this would be welcome.  I can try reverting to an 
>> earlier Git commit tomorrow.
>>
>> Bill
>>
>>  Samples%   Function
>>73203  70% - redisplay_internal (C function)
>>71496  68%  - jit-lock-function
>>71488  68%   - jit-lock-fontify-now
>>71472  68%- jit-lock--run-functions
>>71472  68% - #
>>71472  68%  - font-lock-fontify-region
>>71468  68%   - font-lock-default-fontify-region
>>71436  68%- font-lock-fontify-keywords-region
>>71124  68% - org-do-latex-and-related
>>71124  68%re-search-forward
>>  228   0% + org-activate-folds
>>   16   0% + org-fontify-meta-lines-and-blocks
>>   12   0% + org-activate-footnote-links
>>8   0%   org-activate-dates
>>8   0% + org-activate-links
>>4   0%   org-do-emphasis-faces
>>4   0%   org-fontify-drawers
>>4   0%   org-raise-scripts
>>   28   0%+ font-lock-unfontify-region
>>4   0%+ font-lock-extend-region-wholelines
>>4   0%  text-property-any
>>30951  29% + command-execute
>>   ...
>
> Did you customize org-latex-and-related-regexp ?  Is it still slow if you set
> this to nil (which is apparently the default value) ?
>
> If not, telling which version of Org, that you were using before, might help.

Oops, sorry, the customization option is:

org-highlight-latex-and-related

(the other value is computed by Org)

You may try to change that option org-highlight-latex-and-related to see
if it helps.


Bruno

>
> Bruno
>
>
>>
>> --
>> William Denton
>> https://www.miskatonic.org/
>> Librarian, artist and licensed private investigator.
>> Toronto, Canada

Re: Things got very slow: profiler output

[Bruno Barbier]

Hi William,

William Denton  writes:

> I rebuilt Org and Emacs from the development trees and something is wrong, 
> because some Org files I use regularly have become incredibly slow to use.  I 
> rarely use the profiler and don't know what to make of what it says, but I 
> opened a file and ran it while I moved around and expanded and collapsed some 
> headings for a minute or so.  (It was so slow that it took me a minute to do 
> what would usually take me a few seconds.)  I'll paste the results below.  
> Does that say anything useful?  There is a little LaTeX in the file but not 
> much.
>
> Any help on interpreting this would be welcome.  I can try reverting to an 
> earlier Git commit tomorrow.
>
> Bill
>
>  Samples%   Function
>73203  70% - redisplay_internal (C function)
>71496  68%  - jit-lock-function
>71488  68%   - jit-lock-fontify-now
>71472  68%- jit-lock--run-functions
>71472  68% - #
>71472  68%  - font-lock-fontify-region
>71468  68%   - font-lock-default-fontify-region
>71436  68%- font-lock-fontify-keywords-region
>71124  68% - org-do-latex-and-related
>71124  68%re-search-forward
>  228   0% + org-activate-folds
>   16   0% + org-fontify-meta-lines-and-blocks
>   12   0% + org-activate-footnote-links
>8   0%   org-activate-dates
>8   0% + org-activate-links
>4   0%   org-do-emphasis-faces
>4   0%   org-fontify-drawers
>4   0%   org-raise-scripts
>   28   0%+ font-lock-unfontify-region
>4   0%+ font-lock-extend-region-wholelines
>4   0%  text-property-any
>30951  29% + command-execute
>   ...

Did you customize org-latex-and-related-regexp ?  Is it still slow if you set
this to nil (which is apparently the default value) ?

If not, telling which version of Org, that you were using before, might help.


Bruno


>
> --
> William Denton
> https://www.miskatonic.org/
> Librarian, artist and licensed private investigator.
> Toronto, Canada

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/b

[Bruno Barbier]

Hi Matt,

Matt  writes:

>   On Sat, 24 Feb 2024 17:42:54 +0100  Bruno Barbier 
>
>  > I'll publish a branch soon; it will be a major rewrite of my current
>  > proposal.  It should be less confusing and, I hope, address some of your
>  > comments.
>  > 
>  > Thanks again for your questions and feedbacks,
>
> You're welcome and thanks for sharing your ideas.
>
> Any lack of comments from me recently is just limited time and trying to 
> focus on maintenance.

Thanks for letting me know.

No worries. Take care.

Bruno

Pending contents in org documents (Re: Asynchronous blocks for everything (was Re: ...))

[Bruno Barbier]

Hi,

Bruno Barbier  writes:

> Hi Matt,
[...]
>> Since this thread is dedicated to blocking, let me share my thoughts on that 
>> subject.
>
> I guess I should start a new thread then :)


I finally got a new version.  I've renamed the proposed feature "pending
contents", that is, some contents that something will update *later*.
With that feature, source code blocks and dynamic blocks can be made
asynchronous … and more!

I've publish the proposed changes as a branch.  You can fecth that
branch there:
┌
│ repo:   https://framagit.org/brubar/org-mode-mirror
│ branch: bba-pending-contents
└

The file [my-async-tests.org] describes the proposed changes, its
implementation and contains examples to play with it (shell, python,
ruby, results{append,prepend,silent}, inline blocks, multithreading,
dynamic asynchronous blocks, source codes that depend on other blocks
{post, stdin}, etc.).

I've tried to fully describe the feature in the new section "Pending
contents", in the file `lisp/org-macs.el'.

Testing it in a bare Emacs should now work (I'm using 30.0.50).

*DO NOT TEST* in your production Emacs: this is *alpha* software and it
hooks itself deeply into Emacs (kill hooks), and (only when trying the
multithread examples) uses thread mutexes&locks.

Here is a simple recipe to test the proposed "pending contents" feature:
┌
│ git clone -b bba-pending-contents https://framagit.org/brubar/org-mode-mirror
│ cd org-mode-mirror
│ make compile
│ cd scratch/bba-pending-contents/
│ emacs -q -L ../../lisp my-async-tests.org
└


Thanks you all for your previous comments.  I hope I've addressed most
of them.

Comments, critiques, ideas, corrections are most welcome.

Thanks,

Bruno


[my-async-tests.org]
<https://framagit.org/brubar/org-mode-mirror/-/tree/bba-pending-contents/scratch/bba-pending-contents/my-async-tests.org>

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/b

[Bruno Barbier]

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>>> May you please clarify if adding the new code block parameter that
>>> defines custom execute function is something you want to add to Org mode
>>> or just a helper code to demo you main patch?
>>
>> Yes. (I thought about adding it in a separate request;  but that would be
>> simpler to include it.)
>
> I have doubts about how useful such parameter would be for _users_. It
> is certainly useful for developers of new babel backends, but I do not
> see how users will use it.
>
> I'd prefer to discuss this in a separate thread.

Sure. I'll keep it for an other thread.


>>> Error buffer does not necessarily appear on failure. When the code leads
>>> to process writing to stderr, we also display error buffer. Even if the
>>> process exits with 0 code.
>>
>> Got it.  The current design adds the popup on failure; I should make it
>> more flexible to allow to configure a popup on success too.
>
> By popup, do you mean an overlay where clicking on it will raise the
> usual `org-babel-eval-error-notify' buffer?
> If yes, what happens if a user executes a code block, it fails, and the
> user executes it again without looking at the error? (Second execution
> may be triggered from a different place, indirectly, via noweb ref or
> similar).

Current behavior is to remove the error, and, then re-execute: the
previous errors are gone.  My guess is that this could be improve later,
without breaking the API.


>> But if the execution fails, I guess I'll need to provide some feedbacks
>> anyway, but I don't know yet how (asynchronous popups are not an option,
>> modifying the file neither).
>
> We need something more than just an overlay. Maybe some kind of babel
> execute history (in tabulated-list-mode buffer), keeping information
> about execution stats and stderr data.
>
> We may provide a mode-line or fringe indicator that will warn user if
> something went wrong with recent executions. Something akin compile-mode
> mode line indicator listing errors and warnings during compilation.

I was thinking about using a fringe indicator, for warnings on success (as I'm
removing all overlays on success).

Keeping a buffer of all asynchronous executions looks like a good idea.
I'll think about it.


> (Handling asynchronous messages is actually a complex topic. It has been
> previously discussed on emacs-devel as well. See
> <https://yhetil.org/emacs-devel/838t59j821@gnu.org/>)

I'll look at this.  Thanks!

I'll publish a branch soon; it will be a major rewrite of my current
proposal.  It should be less confusing and, I hope, address some of your
comments.

Thanks again for your questions and feedbacks,

Bruno

>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/b

[Bruno Barbier]

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>> Note that only the first 5 patchs are real patchs.  The remaining things
>> are just a demo how it could be used.  The current async (ob-comint)
>> depends on writing UUIDs in org files, and, that's why I couldn't use it
>> as a demo.  I'm thinking about dropping the patch:
>>
>>'ob-core async: Add org-babel--async tools [2/5]'
>>
>> and just provide an other new keyword (feedbacks-with) so that anybody
>> may plug its own feedback process.
>
> May you please clarify if adding the new code block parameter that
> defines custom execute function is something you want to add to Org mode
> or just a helper code to demo you main patch?

Yes. (I thought about adding it in a separate request;  but that would be
simpler to include it.)


>>> Or consider user running an src block, quickly editing it, and then
>>> running again. What should we do then? Should we stop the first running
>>> process?
>>
>> To keep things simple and generic, I would just forbid the user to
>> reschedule the task until the previous outcome is available.
>
> This may not be as trivial as one might think.
> Consider
>
> #+name: foo
> #+begin_src bash :async yes
> ...
> #+end_src
>
> #+results: foo
> 
>
> Another code block with the same name will write results under
> #+results: foo
> #+name: foo
> #+begin_src bash :async yes
> ...
> #+end_src

> We can currently have multiple code blocks writing to the same place.
> And async execution should not only check is the current src block is
> scheduled, but also whether we are attempting to write to a place where
> no other running block is scheduled to write.

Today, the asynchronous execution is attached to the result; the source
code itself is not locked and stays editable.

With the current implementation, trying your example, the second execution 
fails with:
Error running timer: (user-error "Cannot modify an area being
updated")

So, it's already OK I guess.

I should improve the behavior (to fail before launching the second
execution).  But, the current design should work in that case.


>
>>> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How 
>>> will it interact with asynchronous evaluation? What if we have multiple 
>>> simultaneously arriving error notifications?
>>
>> In the asynchronous case, I've decided to leave the overlay in place
>> in case of errors, with the error description.  Clicking on that error
>> pops up the same kind of popups as in the synchronous case.
>
> Error buffer does not necessarily appear on failure. When the code leads
> to process writing to stderr, we also display error buffer. Even if the
> process exits with 0 code.

Got it.  The current design adds the popup on failure; I should make it
more flexible to allow to configure a popup on success too.


> Also, your code currently creates overlays unconditionally.
> However, when we have :results silent or :results none, Org babel must
> not modify buffer.

Yes. I'll fix it (and that one was in my TODO list :-))

But if the execution fails, I guess I'll need to provide some feedbacks
anyway, but I don't know yet how (asynchronous popups are not an option,
modifying the file neither).

Thanks for your review and questions,

Bruno


>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: org-mode: example blocks are no longer syntax highlighted in emacs

[Bruno Barbier]

Hi Rudy,

Note that I am not an org maintainer; just trying to help :-)

Rudi C  writes:

>> Why is `:eval never` not as good ?  You don't have to write it on each
>> code block; you may set it globally, per file, per headline, etc.
>
> The main reason is that I also use source blocks for babel blocks that are
> runnable. So I cannot use per headline etc. solutions. I can use snippets
> to automatically insert `:eval never`, but it's not as good a UX.

You could customize `org-insert-structure-template' like this:

(add-to-list 'org-structure-template-alist
  (cons "p"  "src python :eval never"))

Then:

C-c C-, p

should insert your code example:

   #+begin_src python :eval never
   #+end_src


> I also already have lots of org-mode notes written using `example lang`
> blocks. Rewriting all of these will be a PITA.

You could teach Emacs to fix/rewrite those for you.  Ask here if you
need help.  You're probably not the only one that will need to fix
them.


> NickD has identified the commit that introduced this breaking change:
>
> ...
> https://github.com/bzg/org-mode/commit/616e80a9f10c4bd085d7b5ac96fd6ea23e9c9191
> I wrote this patch which can be put in one's config to make this behavior
> configurable.

Thanks for the link.

(your code layout was broken, it's more reliable to attach these kinds
of documents)


> Can you merge this upstream? emacs is all about user customizability, and
> in general, breaking backward compatibility without giving a way out is bad.

According to the email thread, the change was made only to match the
documentation (see
https://list.orgmode.org/87zg7lghzz@gmail.com/).

So, maybe a new option, like you suggested will be accepted, something like:

   (defvar org-fontify-block-types '("src")
  "Which block types to fontify.
  Prior versions of Org may fontify other blocks than src.  That
  invalid behavior is now fixed.  If you have documents relying on
  this invalid behavior, and, you cannot fix them, add the block
  types to this list.")
  
But:
   -  Fontification must be fast. The more options, the harder to
  optimize.

   -  It might just be opening a can of worms: Why is editing not
  working too then ? Why is the indentation incorrect? Why are
  they not exported correctly ? Why is pandoc not able to handle
  them correctly ?  etc.

Let see what other people think.

Bruno

Re: org-mode: example blocks are no longer syntax highlighted in emacs

[Bruno Barbier]

Hi Rudi,

Rudi C  writes:

> After upgrading to emacs 29.2 and org 9.7, my example blocks are no longer
> syntax highlighted in emacs. They used to be syntax highlighted when I
> specified the block's language. Any ideas?
>
> I use Doom, so it might have been a third-party feature.
>
> I know org-mode officially suggests that example blocks should not have
> syntax highlighting, but I want it anyway. IMO `:eval never` is just not as
> good,

Why is `:eval never` not as good ?  You don't have to write it on each
code block; you may set it globally, per file, per headline, etc.

See https://orgmode.org/manual/Using-Header-Arguments.html.


> not to mention tools like `pandoc` also produce example blocks from
> markdown source blocks.

My version of pandoc generates org source blocks from markdown code
blocks (pandoc 3.1.11.1).

I'm not sure you'll get a better answer than this SE answer:

  
https://emacs.stackexchange.com/questions/76466/how-do-i-get-syntax-highlighting-in-example-blocks-when-exporting-org-mode-to-ht

I.e. "begin_example" is just not designed for source blocks.  It
may have worked by chance before.

Hoping this helps,

Bruno

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/b

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

>
> Thanks for the code!
> It is a lot more that I expected.

Note that only the first 5 patchs are real patchs.  The remaining things
are just a demo how it could be used.  The current async (ob-comint)
depends on writing UUIDs in org files, and, that's why I couldn't use it
as a demo.  I'm thinking about dropping the patch:

   'ob-core async: Add org-babel--async tools [2/5]'

and just provide an other new keyword (feedbacks-with) so that anybody
may plug its own feedback process.

> I have many questions ;)

Thanks! They are welcome :-)


>> The new API itself is more about how to wait for and display one block
>> result.  So, it's not really aware of sessions.
>
> I think that it is important to think about sessions. For non-sessions,
> we may execute the code in parallel, disregarding already running
> execution. In contrast, for sessions, we need to maintain some queue and
> wait until previous execution finishes before running next (if multiple
> session blocks are executed in quick succession).

I agree.  My proposed patch leaves this open to further work.

My proposed patch allows to have one way, in ob-core, to display
progress of asynchronous executions: like not started, 10%, success,
failed; and a way to plug in asynchronous executions in ob-core.

I've included some demo code just to show how that API could be used.


> It may also be necessary to provide an UI listing the queued and running
> execution. Users should be able to stop "stale" processes if they are
> defunc (consider infinite loop).

That looks nice but that may be too limiting.  Like it will probably forbid
to execute something asynchronously using elisp threads.  Anyway, that
outside the intended scope of my set of patchs.


> What is more important is when users, for example, remove the whole
> subtree containing the place where the execution result is to be
> inserted. Or when users perform edits at or around that place where the
> result is to be inserted. Or consider subtree with pending result
> refiled elsewhere (different file or different place in the same file);
> or archived.

My patch definitely needs to address those, even if only by raising an
error.

> Or consider user running an src block, quickly editing it, and then
> running again. What should we do then? Should we stop the first running
> process?

To keep things simple and generic, I would just forbid the user to
reschedule the task until the previous outcome is available.


> I have several general questions:
>
> - what if append/prepend to result asynchronously?

You mean if org is executing several times the same code concurrently?
I think we should forbid it.

> - what if we replace the result, call async evaluation two times, and they 
> arrive in opposite order (first called is calculated after the second)?

"One execution at a given time" will solve this too :-)

> - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How 
> will it interact with asynchronous evaluation? What if we have multiple 
> simultaneously arriving error notifications?

In the asynchronous case, I've decided to leave the overlay in place
in case of errors, with the error description.  Clicking on that error
pops up the same kind of popups as in the synchronous case.  You can
give it a try using examples of errors my demo file:

   scratch/bba-ob-core-async/my-async-tests.org

(Please use the updated set of patchs that I've sent today, else, you
will have to fix the required libraries).


> Note that we already have a WIP an asynchronous API in the works.
> Check out `org-async-call' in
> https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377
> If possible, we should reuse it.

Thanks. I will have a look at it.


Bruno

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/b

[Bruno Barbier]

Hi Jack,

Jack Kamm  writes:

> Bruno Barbier  writes:
>
>> I'm not using it with official org backends (yet).  I'm using it with
>> several custom backends that I'm working on.  One of the backend
>> delegate the block executions to emacs subprocesses: so I have a kind of
>> asynchronous executions for free for any language, including elisp
>> itself.
>
> For sessions, wouldn't running in a subprocess prevent the user from
> directly interacting with the REPL outside of Org?

Good point. The REPL should be created in the same subprocess; the
REPL display and interaction must happen in the user main emacs.  If
the REPL is based on comint, it should be relatively easy to
implement.


> If so, that's a problem. Org-babel sessions need to play nicely with
> inferior Python, inferior ESS, and other interactive comint modes.

With this solution, the user and the REPL/execution will be in
separate processes; so there will be disavantages.  For basic
interactions, mostly based on text input/output, it should work well.


>> So, here we go.  You'll find attach a set of patchs.  It works for me with
>> Emacsc 30.50 and 9.7-pre (from today).
>
> I suggest to keep these patches on a public branch somewhere, see:
> https://orgmode.org/worg/org-contribute.html#patches
>
>   "When discussing important changes, it is sometimes not so useful to
>   send long and/or numerous patches.
>   
>   In this case, you can maintain your changes on a public branch of a
>   public clone of Org and send a link to the diff between your changes
>   and the latest Org commit that sits in your clone."

Good point.  I'll switch to such a solution as soon as possible.


> I tried running your example on emacs29 using
>
>   emacs -q -L  /path/to/org-mode/lisp my-async-tests.org
>
> but it fails with the error below. Also "make" gives a bunch of
> compilation warnings (which I've put at the bottom).
> ...

My bad: I should have compiled the demo code in a standalone emacs.
I forgot to require some libraries: cl-lib and org-id.

I've now tested with your command line (thanks).  It should now
work. Sorry about that.


> Finally here are the warnings when running "make":
I should have fixed everything; no more (new) warnings.
Thanks!


Please find attached the new set of patchs.  I'll switch to using a
clone and a branch soon, in case if you prefer to wait.

Thanks again!


Bruno



>From f67829454ac0d3cd142da1bd0006efa37acce588 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER 
Date: Fri, 16 Feb 2024 14:31:36 +0100
Subject: [PATCH 1/8] ob-core async: Add faces [1/5]

lisp/org-faces.el (org-async-scheduled, org-async-pending,
org-async-failure): new faces
---
 lisp/org-faces.el | 18 ++
 1 file changed, 18 insertions(+)

diff --git a/lisp/org-faces.el b/lisp/org-faces.el
index 0e20de51a..5a8a8fd51 100644
--- a/lisp/org-faces.el
+++ b/lisp/org-faces.el
@@ -736,6 +736,24 @@ (defface org-mode-line-clock-overrun
   "Face used for clock display for overrun tasks in mode line."
   :group 'org-faces)
 
+(defface org-async-scheduled '((t :inherit org-tag :background "gray"))
+  "Face for babel results for code blocks that are scheduled for execution."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
+(defface org-async-pending '((t :inherit org-checkbox :background "dark orange"))
+  "Face for babel results for code blocks that are running."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
+(defface org-async-failure '((t :inherit org-warning))
+  "Face for babel results for code blocks that have failed."
+  :group 'org-faces
+  :version "27.2"
+  :package-version '(Org . "9.5"))
+
 (provide 'org-faces)
 
 ;;; org-faces.el ends here
-- 
2.43.0

>From 9f135bd5e8e153323bed5a3274851fa78f246b83 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER 
Date: Fri, 16 Feb 2024 14:32:00 +0100
Subject: [PATCH 2/8] ob-core async: Add org-babel--async tools [2/5]

---
 lisp/ob-core.el | 213 
 1 file changed, 213 insertions(+)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index bfeac257b..d98626fe8 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -792,6 +792,219 @@ (defun org-babel-session-buffer (&optional info)
 (when (org-babel-comint-buffer-livep buffer-name)
   buffer-name)))
 
+(defun org-babel--async-status-face (status)
+  (pcase status
+(:scheduled 'org-async-scheduled)
+(:pending   'org-async-pending)
+(:failure   'org-async-failure)
+(:success   nil)
+(_ (error "Not a status"))
+))
+
+(defun org-babel--async-make

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/b

[Bruno Barbier]

Hi Matt,

Thanks for your answer.

Matt  writes:

> ...
> If I understand correctly, there are several independent topics the code 
> addresses:
>
> | topic| manner addressed   |
> |--+|
> | execution status | using overlays to communicate execution status |
> | locating results | using overlays to locate results   |
> | blocking | making all execution asynchronous  |
> |--+|
>
> I suggest these be discussed in separate threads.

Actually my patch is only about reporting execution status and
inserting results, in a generic way, in ob-core.  I provided some demo
code on top of them, just to show how it could be used.

Sorry about the confusion.  

> ...
>
> Since this thread is dedicated to blocking, let me share my thoughts on that 
> subject.

I guess I should start a new thread then :)


> ...
>
>
> Executing a shell block requires starting a 
> [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html][process]].
>
> Processes are synchronous or asynchronous.
>
> Three primitives exist in Emacs for making processes:
>
> 1. make-process (asynchronous)
> 2. call-process (synchronous)
> 3. call-process-region (synchronous)
>
> There exist several convenience wrappers for these.  AFAIK, everything 
> reduces to these three primitives.  For example, =async-shell-command= runs 
> =call-process= and prevents blocking by appending a "&" to the command which 
> tells the shell to run the command in the background and return control to 
> the terminal.  This background-foreground distinction is called "job control".
>

> Output from a process typically goes to a buffer.  This may be changed and 
> instead handle output with a filter function.  =call-process= has an option 
> to directly send output to a file.
>

I've reached the same conclusion. As I wanted one simple
implementation, I was thinking about using 'make-process', but, as we
have to deal with the output manually and use a process sentinel, and,
as offering a REPL to the user is nice, my personal pick is to use a
plain shell with comint.

> Subprocesses inherent the =default-directory= and the environment from Emacs. 
>  The environment may be changed using =process-environment=.
>

One wonderful thing about using =default-directory= is that, if done
right, tramp will automatically handle executions on one or more remote
computers.

> There are two types of asynchronous connections: "pty" ("pseudoterminal") and 
> "pipe".  The main difference is that "pty" provides a terminal-like 
> connection which allows for things like job control (=C-c=, =C-z=, etc.).

I'm personally ignoring the difference between pty and pipe: as I
don't use the terminal features, it doesn't really matter to me.


> ...
> I'm not sure how we could make a persistent, synchronous process.  
> Persistence is achieved, currently, by a process buffer.  Is there another 
> way persistence may be achieved?

If I understand correcly, "persistent" here means to preserve some
state between executions.  They are definitely other way to preserve
and provide a state.

If you focus on processes only, it's probably OK. But, if you use
threads or some other kinds of asynchronicity, that might be too
limiting to require a buffer and a process.


> Of course, this ignores whether a persistent, synchronous process is even 
> desirable.  Given reliable asynchronous execution with persistence, I can't 
> think of reason why someone would prefer a blocking operation.
>

I'm not so sure.  If the success of an execution is really important,
and, intrinsically unreliable (like a network connection), and, it
only takes a few seconds, I'll probably choose synchronous execution,
just to see that everything is OK, before moving to my next task.



> ...
> Attached are two files, ob-blub.el and ob-blub-test.org.  Download both to 
> the same directory.  Run the first block in ob-blub-test.org.  This imports 
> ob-blub, loads it into Babel, and sets up blub to be whatever 
> =shell-file-name= is (for example, bash).  If you want to try Python or Ruby, 
> comment out the shell configuration, uncomment the Python or Ruby 
> implementations, and evaluate the block again.  Hopefully ob-blub.el is 
> documented sufficiently for you to experiment.

Thanks for sharing.  I didn't have time to try it yet.  But it looks
like you could use the 'execute-with' keyword that my patchs provide.
That way, you may redirect evaluation to use your "blub"
implementation, and still use the real language name for your code
blocks (that way, org knows how to fontify them, edit them, etc).  And, you
will not have to manually modify and reevaluate your elisp when
switching languages.

> The blub implementation has the same shortcomings, at least for shells, as 
> the current shell implementation.  It has a few ideas, su

Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/b

[Bruno Barbier]

Hi Matt, Jack, Ihor,

Sorry for the late reply.  Cleaning the code took me longer than
expected.


Jack Kamm  writes:

> Bruno Barbier  writes:
>
>> FWIW, I've been trying to use asynchronous blocks for everything, not
>> only the source blocks that are based on the comint mode.
>> ...
>
> Sounds interesting, a couple questions:
>
> 1. Which languages have you implemented/tested this on?

I'm not using it with official org backends (yet).  I'm using it with
several custom backends that I'm working on.  One of the backend
delegate the block executions to emacs subprocesses: so I have a kind of
asynchronous executions for free for any language, including elisp
itself.


> 2. Does it apply for sessions, nonsessions, or both?
>

The new API itself is more about how to wait for and display one block
result.  So, it's not really aware of sessions.

But, I usually try to think as "no session" as a "one shot" session
(like Matt wrote in his email).  So, in that sense, it works for both
anyway ;-)


> Also interesting, I think it's worth exploring/testing this overlay idea
> out. Does that mean that output is asynchronously printing into the Org
> buffer? It sounds cool but I wonder if it might cause problems while
> trying to edit another part of the buffer.

Currently, I've limited the progress feedback to fit on one line to
avoid anoying vertical display jumps.  When the computation is
successful, the result is inserted as usual, and, that may be annoying;
when it updates a previous result of the same height, it's ok.  Else, we
could fold it to stay on one line too (using the overlay), until the
user explicitly request to see it.

Jack Kamm  writes:
>
> That all sounds reasonable...if you work on this, let me know if you
> want any help with testing.

Thanks.  I'll definitely appreciate your help, to test the current
code with the Python backend or any other backend if you prefer.


Matt  writes:

> If I followed correctly, the topic switched to discussing async
> generally within Babel.  I've started a new thread accordingly.

Indeed. Thank you!


>  > FWIW, I've been trying to use asynchronous blocks for everything, not only 
> the source blocks that are based on the comint mode.
>
> I've been trying to figure out how to make everything async by default.  I'm 
> super interested to see what you've come up with.

Good to know.  Let's we make this happen! 


>  > I think it would be good if ob-core itself could provide an asynchronous 
> API.
>
> Fortunately or unfortunately, depending on how you look at it, Babel already 
> does.
>

> The challenge is that the Org Babel API has grown piecemeal over 14 years.  
> It's been written by several authors with limited time and knowledge.  The 
> result, while powerful and useful, is a bit of a hodgepodge.  A prime example 
> is that the concepts of "persistence" and "synchronicity" are conflated.  
> "Session" is often used to mean "asynchronous" even though the two ideas are 
> orthogonal.  Emacs provides primitives that could make non-session blocks 
> asynchronous.  It's historical accident that blocks aren't async by default.

> For me, the issue is that the Babel API needs some high level perspective in 
> order to make it consistent.

> I see the following terms as guides.  If we can separate these concepts 
> within the API, then Babel to *feel* like an API:
>
> - "Session" means a shell environment is "persistent."  Each call is executed 
> in the same environment.  State exists between calls.
>
> - "Non-session" means a shell environment is "temporary."  Each call is 
> executed in an independent environment.  State does not exist between calls.
>
> - "Synchronous" means that execution prevents the user from editing the 
> document while results are obtained.
>
> - "Asynchronous" means that execution does not prevent the user from editing 
> the document while results are obtained.

I mostly think the same.  Sessions (including the "none" session)
definitely need some generic API and some generic tests that all
backends could just reuse.

To execute Python blocks, using the proposed async API:

   - I've (re)implemented the "asynchronous with session" case (copying/pasting
 the relevant part from ob-python).
   
   - The "synchronous case" is just artificially blocking the user until
 the asynchronous result is known (which looks incredibly tricky to
 implement if even possible...).
   
   - The "no session" case is just about creating a new unique session
 and throwing it away immediately.

But, some users may rely 

Re: Async Python src block behavior with :dir header property

[Bruno Barbier]

Ihor Radchenko  writes:

> Jack Kamm  writes:
>
>> On executing any python session block I am getting the following error
>> which I think is caused by the above:
>>
>>   Debugger entered--Lisp error: (void-variable buffer-name)
>
> That's a mystery.

It looks like 'when-let*' doesn't accept symbols without values; it
needs real bindings.  The form `let*' assigns nil when there are no
values, but, with `when-let*', assigning nil would make the whole
`when-let' fails.

Here is the expansion when using a symbol without a value, which
explains both the void-variable error and the compiler message:

   #+begin_src elisp :results scalar
 (macroexpand-all '(when-let* (new-var) 7))
   #+end_src

   #+RESULTS:
   : (let* ((new-var (and t new-var))) (if new-var 7))



Bruno

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Bruno Barbier]

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>> While the execution is pending, I'm using the same technique that Org is
>> using when a source block is being edited: the result is left untouched,
>> but below an overlay.  The overlay is used to know where to insert the
>> result and to display the status/progress of the execution.  If the file
>> is closed and the execution fails, nothing is lost, the old result is
>> still available.
>>
>> If that technique looks safe enough and interesting, I can prepare a set
>> of patches so that we can discuss it further and, maybe, add it in Org.
>
> Using overlay also sounds good.
>
> I am wondering what you do when there is no result yet or when user
> edits whatever is under the overlay,
> but that's just a technical detail.

I don't think it's really robust yet. We'll see how to improve/change
it.

I'll prepare the set of patchs.


Bruno


>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Bruno Barbier]

Hi Ihor, Jack,

Ihor Radchenko  writes:

> Jack Kamm  writes:
>

>> I agree that it would be good to redesign it, but am not sure where to
>> start.
>
> For example,
>
> 1. Change `org-babel-comint-async-register' to return UUID and to store
>PARAMS as passed by the backend (current approach with PARAMS being
>derived from src blocks prevents backends to transform src block
>PARAMS dynamically).
> 2. Change `org-babel-insert-result' to handle :async t specially,
>inserting something reliable, like #+async:  in place of result
>without performing extra transformations.
> 3. Change `org-babel-insert-result' to accept an internal parameter
>that will make it replace #+async:  keyword rather than perform
>normal result insertion.
> 4. Change `org-babel-comint-async-filter' to use the previously passed
>PARAMS, remove :async t from them, and arrange the call to
>`org-babel-insert-result' to replace the #+async:  keyword.
>

FWIW, I've been trying to use asynchronous blocks for everything, not
only the source blocks that are based on the comint mode.  I think it
would be good if ob-core itself could provide an asynchronous API.  I've
modified my Org so that it does have such an API.  This is work in
progress; let me describe it.

I've modified ob-core itself to allow asynchronicity.  In the
asynchrosous case, instead of calling:

  (org-babel-execute:LANG body params)

I'm calling:

  (org-babel-schedule:LANG body params handle-result)

where `org-babel-schedule:LANG' is in charge of calling `handle-result'
with the result (or the error) when it is known; `handle-result' takes
care to call `org-babel-insert-result' at the correct place (and
`org-babel-insert-result' is only called with a real result).

While the execution is pending, I'm using the same technique that Org is
using when a source block is being edited: the result is left untouched,
but below an overlay.  The overlay is used to know where to insert the
result and to display the status/progress of the execution.  If the file
is closed and the execution fails, nothing is lost, the old result is
still available.

If that technique looks safe enough and interesting, I can prepare a set
of patches so that we can discuss it further and, maybe, add it in Org.

Let me know,
Thanks,

Bruno

Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

>
> This is most likely something about my current system setup - I can
> reproduce with other Org mode and Emacs versions. But I have no clue
> what is the cause.

I'm getting the same as you with your MWE.
   
The tag, used by ob-comint async, is:

   "/tmp/babel-zqh04P/python-GL5N5d"
   
but, in "/tmp/bug.org" it becomes:

   "babel-zqh04P/python-GL5N5d"

(`org-babel-result-to-file' transformed it into a simpler relative
path).

The filter `org-babel-comint-async-filter' cannot spot it, because
it's searching for the exact string "/tmp/babel-zqh04P/python-tXsdFw".

Here is how to reproduce:
   #+begin_src elisp :results table
 (let* ((tag "/tmp/babel-zqh04P/python-tXsdFw")
(repro 
 (lambda (fn)
   (let ((lnk 
  (with-temp-buffer
(org-mode)
(let* ((default-directory "/tmp")
   (buffer-file-name  fn)
   (cbuf (clone-indirect-buffer "tmp" nil)))
  (with-current-buffer cbuf
(org-babel-result-to-file tag))
 (list fn
   (not (eq nil (string-match-p (regexp-quote tag) lnk)))
   lnk)
   (cons (list "Filename" "string-match-p" "Org link")
 (cons 'hline
   (mapcar repro (list "/tmp/bug.org" 
   "/somewhere/else/bug.org")
   #+end_src

   #+RESULTS:
   | Filename| match-p | Org link   
   |
   
|-+-+---|
   | /tmp/bug.org| nil | [[file:babel-zqh04P/python-tXsdFw]]
   |
   | /somewhere/else/bug.org | t   | 
[[file:../../tmp/babel-zqh04P/python-tXsdFw]] |


I don't know what a proper fix would be though.
   
Hoping this help,

Bruno


>
> Emacs  : GNU Emacs 30.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version 
> 3.24.39, cairo version 1.18.0)
>  of 2024-01-30
> Package: Org mode version 9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ 
> /home/yantar92/.emacs.d/straight/build/org/)
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at .
> Support Org development at ,
> or support my work at 

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
>> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
>> multiple lines, getting the same results as when I type them manually,
>> or copy them line by line. My xterm doesn't seem to allow me to do that.
>
> The behavior of M-x shell can indeed be made use of.
> However, this particular difference with xterm, AFAIU, is not
> documented - unaware users may be surprised.
> The situation is worse with Org shell blocks - users naturally expect
> script-like behavior (even for :session), but run into edge cases like
> this and get confused.
>
> We should either document the caveats, or, preferably, make the behavior
> more consistent with expectations. At least, by default.

> That's why I think that filing a bug report makes sense from Org mode
> project point of view.

Thanks for the explanation.  I see you got my point.  We'll see what
Emacs maintainers will say about the current behavior of M-x shell;
filling the bug report was definitely a good idea anyway.



Ihor Radchenko  writes:
> __By default__, Org should produce more expected behavior - what users
> would get from running a script file rather than from redirecting stdin.
> We can optionally leave the stdin redirection as an option to be used by
> the users who understand the peculiarities.

I agree that it would be simpler to switch to the script-like behavior
by default on org side.  About the interactive-like behavior, that
would be nice to keep it, if some people rely on it in their existing
documents (I personally don't).

Bruno

Re: Non-emacs shell (Re: bash source code block: problem after ssh commands)

[Bruno Barbier]

Hi Max,

Max Nikulin  writes:

> On 17/11/2023 22:47, Bruno Barbier wrote:
>> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
>> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
>> multiple lines, getting the same results as when I type them manually,
>> or copy them line by line. My xterm doesn't seem to allow me to do that.
>
> I am unsure what do you expect from xterm, but perhaps it is not 
> responsibility of a terminal application.

It has been said in this thread that 'M-x shell' should be fixed to
match the behavior that we see in a plain terminal, when we copy
multiple lines.  I just wanted to point out that I do prefer the way
'M-x shell' handles the copy of multiple lines.


> Multiple lines can be copied to regular BASH prompt (bracketed paste is 
> enabled by default nowadays), however it may be inconvenient to edit.
> You may use edit-and-execute-command (C-x C-e) (BASH, not Emacs key 
> binding) to start an editor and to paste multiple commands there.
> See also "fc" BASH built-in for editing and re-executing last commands.

Thanks Max! I didn't know that.  I should definitely start using this
when I'm stuck in a console, to safely copy/edit my commands using
Emacs.

Thanks,


Bruno

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Bruno Barbier]

Matt  writes:

>   On Sat, 18 Nov 2023 09:54:46 +0100  Bruno Barbier 
>  
> It's still not clear to me if this is "what Emacs does".  However, that's the 
> best I could come up with.  
>
> Evaluating the following 
>
> #+name: /tmp/test.sh
> #+begin_src bash :results output
> ssh localhost "echo foo>foo_file"
> echo "bar" | tee /tmp/bar.txt
> #+end_src
>
> does exactly what
>
> cat /tmp/test.sh | bash -c bash
>
> does.  Both create a file "foo_file" containing "foo" on the remote machine 
> and neither execute the second line.
>
> So, I would say that what happens in Org is the "expected" behavior.

Agreed.

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Bruno Barbier]

Matt  writes:

>   On Sat, 18 Nov 2023 09:29:56 +0100  Bruno Barbier 
>
>  > IIUC, what Max is saying is that you should not concentrate on
>  > *that specific command* because that command doesn't do what you think
>  > it does.
>
> Cool, it sounds like we're agreed (albeit for different reasons).

Great :-)


>  > To reproduce, I'm personally still using:
>  > 
>  > cat /tmp/test.sh | bash
>  > 
>  > which is, IIUC, what:
>  > 
>  > (process-file "bash" "/tmp/test.sh")
>
> Yes, agreed.  I think that's more like what's happening.  
>
> What about the ("-c" "bash") passed into process-file?  

Useless indirection when the command is "bash" is the same as
'shell-file-name', like in our case ?

Maybe ...

But, you're right.  To be safe, from now on, I'll use:

cat /tmp/test.sh | bash -c bash


Thanks.

Bruno

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Max Nikulin  writes:

> On 25/10/2023 18:17, alain.coch...@unistra.fr wrote:
>> By contrast, it works with this one:
>> 
>> #+begin_src bash :results output
>> sshcoch...@fruc.u-strasbg.fr  "echo foo>foo_file" ; echo "bar"
>> #+end_src
>
> What about
>
> #+begin_src bash :results output
> ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
> echo more
> #+end_src
>
> ?

For me:

 #+begin_src bash :results output
 ssh phone "echo foo>foo_file" ; echo "bar"
 echo more
 #+end_src

 #+RESULTS:
 : bar

And, telling SSH to not swallow the remaining commands (option '-n'):

 #+begin_src bash :results output
 ssh -n phone "echo foo>foo_file" ; echo "bar"
 echo more
 #+end_src

#+RESULTS:
: bar
: more


Bruno

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Bruno Barbier]

Matt  writes:

>   On Sat, 18 Nov 2023 04:11:03 +0100  Max Nikulin  wrote --- 
>
>  > >  bash -c bash /tmp/two-lines.sh
>  > 
>  >  From my point of view it was a plain mistake in attempts to simulate 
>  > the issue outside of Emacs. There is no point to concentrate on this 
>  > command. I tried to explain that it is incorrect usage of "-c" shell 
>  > option and what is the actual effect of this call, but I seems I failed.
>
> As an ob-shell user, my expectation is that execution within Org produces the 
> same behavior as outside of Emacs.  This is why I've focused on the command.  
> It acts as a guide for what is "correct."  Maybe this is misguided for a 
> reason I don't yet see?

IIUC, what Max is saying is that you should not concentrate on
*that specific command* because that command doesn't do what you think
it does.


To reproduce, I'm personally still using:

cat /tmp/test.sh | bash

which is, IIUC, what:

(process-file "bash" "/tmp/test.sh")

is doing, that is:

   The program’s input comes from file INFILE


Bruno

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Hi Matt,

Thanks this summary and for working on this!

Just a few comments/corrections about some specific points, hoping it
might help.

Matt  writes:

>   On Fri, 17 Nov 2023 10:20:28 +0100  Ihor Radchenko  wrote --- 
>
>  > This has nothing to do with Emacs comint and this is also not a bug in
>  > Emacs 
>
> Ihor, there were two claims made in the original report.  I was referring to 
> Claim 2.  That deals with M-x shell and therefore comint-mode.
>
> Regarding Claim 1:
>
> - Can anyone verify Claim 1?

I do: the file is created and the command "echo bar" is NOT executed.

Here is my code block and its results:

#+begin_src bash :results output
  ssh phone "echo foo>foo_file"
  echo "bar"
#+end_src

#+RESULTS:

No results (the echo command is NOT executed).

The file "foo_file" is created on the remote; its content is "foo".

#+begin_src bash :results output
  date
  ssh -n phone "ls -alh foo_file"
  ssh -n phone "cat foo_file"
#+end_src

#+RESULTS:
: Sat Nov 18 08:33:59 CET 2023
: -rw--- 1 u0_a256 u0_a256 4 Nov 18 08:26 foo_file
: foo



> - What versions are people using?
>   + M-x org-version
>   + M-x emacs-version

#+begin_src elisp
  (list emacs-version org-version)
#+end_src

#+RESULTS:
| 30.0.50 | 9.7-pre |

GNU/Linux gentoo

> ...

> * Comments about the claims:

> ** Comment 1.
> ...
> I am unable to reproduce the reported behavior (of
> "bar" not returning).  Instead, I get an ssh-askpass permission denied
> error, foo_file is not created, and "bar" is given as the result.  I
> do not see anywhere in the thread that the original claim was
> reproduced.

It seems your SSH failed to connect.  In that case, I cannot swallow the
second command; thus the command "echo bar" is executed.

I can reproduce what you see on my side if I force the connection to fail:

#+begin_src bash :results output
  ssh WRONG_REMOTE "echo foo>foo_file"
  echo "bar"
#+end_src

#+RESULTS:
: bar


>
> The thread preceded something like follows.
>
> Leo Butler suggested two work arounds:
>
> - add the -f to the ssh command


> - add a semi-colon and line continuation to the first line.
>
> Russell Adams suggested another work around:
>
> - add -n to the ssh command

That's the one I use; the option -n is enough for me ('-n' = Redirects
stdin from /dev/null). The option '-f' means SSH will go to background;
I'm not sure I want that.

> ...

> ... 
> He then proposes an experiment to close stdin.  To do this, he calls
>
> #+begin_src shell :results output
> exec 0>&-
> echo OK
> #+end_src
>
> He claims that "exec 0<&-" closes stdin.  I believe there is a typo.
> ...

You're right. Good catch, thanks!

Although it seems to work either way on my side.

#+begin_src shell :results output
  exec 0<&-
  echo OK
#+end_src

#+RESULTS:

#+begin_src shell :results output
  exec 0>&-
  echo OK
#+end_src

#+RESULTS:


> What Bruno writes corresponds to "closing output file descriptor 0".  I 
> honestly don't know what the difference is between an "output file 
> descriptor" and an "input file descriptor".  I had no luck finding this 
> information in man bash or info bash.
>

My point was: the commands are read the standard input, thus, any
command that modifies that standard input will modify what gets
executed.


> ...
> This is what we see in Org.  I'll be honest, though, I don't
> really know what to expect with exec 0>&- and exec 0<&-.  When I call
> them in the terminal, it kills the terminal.

Let's forget about 'exec 0<&-' (closing the standard input/outputs):
this is bringing other corner cases.  But, yes, I would expect a
terminal to close itself automatically if its input is closed.

> ...
> As far as I can tell, though, that's not what prevents "bar" from being 
> returned.  As far as I can reproduce, calling
>
> #+begin_src bash :results output
> ssh localhost "echo foo>foo_file"
> echo "bar"
> #+end_src
>
> *does* give "bar" for results even though it shouldn't.

Does it echo bar when the SSH connection succeeds too ?


Thanks again for working on this.


Bruno

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Hi Matt, Ihor, Alain,

Ihor Radchenko  writes:

> alain.coch...@unistra.fr writes:
>
>> At the most basic user level (i.e., non lisp aware), why is it not
>> necessarily a bug if "something" does the expected in an X terminal
>> but not in an emacs terminal?  I think Matt and I (and others) are on
>> the same page here, and he already essentially drafted the bug report
>> I would have sent (only better).
>
> WRT M-x shell, feel free to submit a bug report. I mostly pointed that
> the problem with M-x shell is not the problem you originally ran to. It
> is a different problem (also, we ran into it in the past).

FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
multiple lines, getting the same results as when I type them manually,
or copy them line by line. My xterm doesn't seem to allow me to do that.


>> What also confuses me is that it seems to me that, in the minimum
>> working example, you consider the ssh command and the read command as
>> equivalent.  But I don't even enter the password when using ssh...

I've introduced the 'read' example, as a simpler way to modify the
standard input and get the same kind of "unexpected" results, without
relying on SSH.  It seems that I have only caused confusion, sorry about
that.

IIUC, the OP example is not working because SSH is modifying the
standard input (with or without passwords).

> I was only able to reproduce your problem with ssh asking a password.
> We are discussing the reproduced case.
>
> If you see problems with
>#+begin_src bash :results output
>ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
>echo "bar"
>#+end_src
>
> even when ssh does not ask for a password, please provide more detailed
> reproducer that we can replicate locally without guessing your ssh config.
>

I'm not the OP, but, my SSH is configured to work without passwords and
SSH is still consuming lines from standard input:

#+begin_src bash :results output
ssh phone echo "remote"
echo "local"
#+end_src

#+RESULTS:
: remote

It looks like it is a known SSH "feature" (see
https://unix.stackexchange.com/a/688024):

#+begin_src bash :results output
seq 100 | (ssh phone sleep 1; wc -l)
#+end_src

#+RESULTS:
: 675173

Same block, but asking SSH to not use stdin (using '-n' as mentionned in
this thread):

#+begin_src bash :results output
seq 100 | (ssh -n phone sleep 1; wc -l)
#+end_src

#+RESULTS:
: 100

IMHO, what ob-shell is doing today seems a valid way of evaluating
source blocks (and it seems to have been like that for a long time).  It
should probably be documented somewhere, so that users know how to write
their source blocks, or switch to another way, like adding a :cmdline
parameter as mentionned in this thread.


Hoping I didn't increase the confusion again,
:-)


Bruno




> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at .
> Support Org development at ,
> or support my work at 

Re: how to exclude certain row values from calculation in a table

[Bruno Barbier]

Hi Uwe,

Uwe Brauer  writes:

> Hi all
>
> Please look at 
>
> |   |  Price |
> |---+|
> | / |  10.98 |
> |   |  11.90 |
> |   |  19.98 |
> |   |  13.79 |
> |   |  29.97 |
> |   |  18.98 |
> |   |  13.79 |
> |   |  11.90 |
> |   |  24.28 |
> |---+|
> |   | 155.57 |
> #+TBLFM: @11$2=vsum(@I$2..@II$2);f2
>
> I would like to exclude the first value from the sum by adding some mark
> in that row.
...
>
> Any idea how to exclude row values in a column?

You could use 'vmask' to mask the values you don't want.
Here is an example using vmask:

   ||  Price |
   |+|
   | 1  |  10.98 |
   ||  11.90 |
   ||  19.98 |
   ||  13.79 |
   | 1  |  29.97 |
   ||  18.98 |
   ||  13.79 |
   ||  11.90 |
   ||  24.28 |
   |+|
   || 155.57 |
   | Total unmarked | 114.62 |
   | Total marked   |  40.95 |
   | Check all  | OK |
   #+TBLFM: @11$2=vsum(@I$2..@II$2);f2
   #+TBLFM: @12$2=vsum(vmask(map(<0 = #1>, @I$1..@II$1), @I$2..@II$2));f2NE
   #+TBLFM: @13$2=vsum(vmask(map(<1 = #1>, @I$1..@II$1), @I$2..@II$2));f2NE
   #+TBLFM: @14$2=string(if(@12$2+@13$2 = @11$2, "OK", "Oops!"))

Bruno

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Hi,

alain.coch...@unistra.fr writes:

> Ihor Radchenko writes on Thu 26 Oct 2023 13:44:
>
>  > I can now reproduce the problem locally.
>  > 
>  > It boils down to
>  > 
>  > (setq exit-status
>  >  (process-file shell-file-name input-file
>  >(if error-file
>  >(list t error-file)
>  >  t)
>  >nil shell-command-switch command))
>  > 
>  > that is an equivalent of
>  > 
>  > bash -c bash /path/to/file-containing-the-source-code.sh
>  > 
...
> I am confused about what you specifically do to "evaluate the above".
> To start with, I have to use quotes to make your command be performed:
>
>bash -c "bash /path/to/file-containing-the-source-code.sh"
>
...
>
> But most importantly, the second line *is* produced, either if I use
> an SSH key for passwordless access or if I enter the password
> manually.

IIUC, the elisp expression:

(process-file "bash" "/tmp/test.sh")

is more equivalent to:

cat /tmp/test.sh | bash

i.e. the shell is getting the commands from stdin.  Thus, any command
that uses stdin might change what gets executed or not.

I'm able to reproduce using the following minimal script, without
passwords nor SSH (where the cryptic first line closes stdin).

#+begin_src shell :results output
  exec 0>&-
  echo OK
#+end_src

The result is "OK" only when commenting out the first line; else, the
echo command is not executed (because stdin has been closed).

Here is an other example, where the second echo is eaten by the script
itself:

#+begin_src shell :results output
  echo 1
  read -p "Next command? " NEXT_COMMAND
  echo 2
  echo 3
#+end_src

#+RESULTS:
: 1
: 3

Bruno

Re: the opposite of the noexport tag

[Bruno Barbier]

Uwe Brauer  writes:

>> Uwe Brauer  writes:
>
>
>> I just tested your example. It works for me, exporting to HTML.  I'm
>> using org version 9.7-pre.
>
>> What is the error ?
>
> Well since I did not set a list, org mode complained about it. Not in
> your case? I exported to LaTeX but it should make any difference


My mistake: I didn't reload the file after adding the local
variables: your example didn't work for me either. Sorry.

Just follow what Ihor told you, it's simpler.


About exporting the tags, the following option seem to do the trick:

#+OPTIONS: tags:nil


See (info "(org) Export Settings").


Bruno.




>> Unrelated, but you don't need the "Local Variables" if you didn't change
>> them elsewhere, as you are using their default values.
>
> I tried now 
> ,
> | #+options: SELECT_TAGS: noexport
> | #+options: EXCLUDE_TAGS: export
> | 
> | Some test
> | * First section
> | 
> | * Second 
> :export:
> | This 
> | * third 
> | 
> | That
> `
>
>
> which is almost what I want besides the fact that the tag itself also
> gets exported,
>
> ,
> | \section{Second\hfill{}\textsc{export}}
> | \label{sec:orgfba34a2}
> | This 
> `
>
>
> Do you say you get the same without changing anything? So
> your headers without a tag don't get exported, hm that seems odd to me.
>
>
> Uwe 
>
>> Bruno
>
>
>
>
> -- 
> Warning: Content may be disturbing to some audiences
> I strongly condemn Hamas bestialic terroristic attack on Israel, especially 
> the despicable pogroms.
> I strongly condemn Putin's war of aggression against Ukraine.
> I support to deliver weapons to Ukraine's military. 
> I support the NATO membership of Ukraine.
> I support the EU membership of Ukraine. 
> https://addons.thunderbird.net/en-US/thunderbird/addon/gmail-conversation-view/

Re: the opposite of the noexport tag

[Bruno Barbier]

Uwe Brauer  writes:

>>>> "BB" == Bruno Barbier  writes:
>
>> Did you try org-export-select-tags ?
>
>> (I didn't but, from its documentation, it might be what you're looking
>> for).
> ,
> | 
> | Yes I tried the following 
> | Some test
> | * First section
> | * Second 
> :export:
> | This 
> | 
> | 
> | # Local Variables:
> | # mode: org
> | # org-export-select-tags: noexport
> | # org-export-exclude-tags: export
> | # End:
> `
>
> But when try export I obtained an error.


I just tested your example. It works for me, exporting to HTML.  I'm
using org version 9.7-pre.

What is the error ?


Unrelated, but you don't need the "Local Variables" if you didn't change
them elsewhere, as you are using their default values.

Bruno

>
> Uwe 
>
>
>
> -- 
> Warning: Content may be disturbing to some audiences
> I strongly condemn Hamas bestialic terroristic attack on Israel, especially 
> the despicable pogroms.
> I strongly condemn Putin's war of aggression against Ukraine.
> I support to deliver weapons to Ukraine's military. 
> I support the NATO membership of Ukraine.
> I support the EU membership of Ukraine. 
> https://addons.thunderbird.net/en-US/thunderbird/addon/gmail-conversation-view/

Re: the opposite of the noexport tag

[Bruno Barbier]

Hi,

Uwe Brauer  writes:

>
> So it would be very handy to configure the exporter for that file
> locally that he only exports sections that have a export tag
>
> Is this possible?

Did you try org-export-select-tags ?

(I didn't but, from its documentation, it might be what you're looking
for).

Bruno.

Re: [the cryptic @@#$7]

[Bruno Barbier]

Hi Uwe,

Uwe Brauer  writes:
> Here is an example where the org-lookup-first method seems to fail

> #+TBLFM: $3='(org-lookup-first $2 '(remote(Table1A, @I$1..@II$1)) 
> '(remote(Table1A, @I$7..@II$7)))::$4='(org-lookup-first $2 '(remote(Table2A, 
> @I$1..@II$1)) '(remote(Table2A, @I$7..@II$7)))::$5=vsum($3..$4);f2

You forgot to update the column numbers: the names to look for are now in
the column '2', and, the totals are now in column '8'. This should work:


#+TBLFM: $3='(org-lookup-first $2 '(remote(Table1A, @I$2..@II$2)) 
'(remote(Table1A, @I$8..@II$8)))
#+TBLFM: $4='(org-lookup-first $2 '(remote(Table2A, @I$2..@II$2)) 
'(remote(Table2A, @I$8..@II$8)))
#+TBLFM: $5=vsum($3..$4);f2


Bruno

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[Bruno Barbier]

Hi Ye, Ihor, Max,


The change seems to come from this commit:

   commit d7a55bbd537314d2776b082bd92a1a08b3edc84e
   Date:   Wed Sep 28 12:07:14 2022 +0800
   org-latex-export-to-latex: Do not suppress major modes in babel

It replaces 'write-file' with 'write-region', but, according to the
documentation of 'require-final-newline', 'write-region' ignores
'require-final-newline'.

See attached diff that reverts the problematic change.

In summary, before this commit, Emacs was fixing exported text files
(HTML, markdown, etc.) for free, adding a newline if needed for text
files.  It's not anymore.  I'm not sure what would be the best way to
fix this though.



Bruno

diff --git a/lisp/ox.el b/lisp/ox.el
index f8ccd2a9f..5ff105d3b 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -6893,7 +6893,7 @@ (defun org-export-to-file
   (with-temp-buffer
 (insert output)
 (let ((coding-system-for-write encoding))
-	  (write-region nil nil file)))
+	  (write-file file)))
   (when (and (org-export--copy-to-kill-ring-p) (org-string-nw-p output))
 (org-kill-new output))
   ;; Get proper return value.

Re: [the cryptic @@#$7] (was: equivalent of VLOOKUP (in ods) to org-table)

[Bruno Barbier]

Uwe Brauer  writes:

>>>> "UB" == Uwe Brauer  writes:
>
>>>> "BB" == Bruno Barbier  writes:
>>> Hi Uwe,
>
>>> Uwe Brauer  writes:
>>>> so the question is what is equivalent of VLOOKUP in org.
>
>>> Did you check these lookup functions in the Org manual?
>
>>> (info "(org) Lookup functions")
>
>
> I ask differently why does the following solution not work
>
> #+Name: table1
> | Name   | Ex1 | Ex2 | Ex2 | Ex4 | Ex5 | ResSh1 |
> |+-+-+-+-+-+|
> | Smith  |   2 |   3 |   4 |   6 |   7 | 22 |
> | Miller |   2 |  10 |   1 |   1 |   5 | 19 |
> | Wick   |   1 |   2 |   3 |  10 |   2 | 18 |
> #+TBLFM: $7=vsum($2..$6);f2
>
>
>
> #+Name: final
> | Name   | Some | ResSh1   |
> |+--+--|
> | Smith  |4 | [22, 19, 18] |
> | Miller |4 | [22, 19, 18] |
> | Wick   |4 | [22, 19, 18] |
> |+--+--|
> #+TBLFM: @2$3..@>$3=remote(table1,@2$7..@>$7)

The ref "remote(table1,@2$7..@>$7)" targets a range, containing all fields of
colum 7.  So, for each field, you get the same value: an array
containing all the field values: [22, 19, 19].


>
> But this solution does, what does @@#$7 mean?

See (info "(org) References"), in the section:
   "Field coordinates in formulas"

"‘@#’ is substituted with the row number of the field where the
current result will go to".


> #+Name: final2
> | Name   | Some | ResSh1 |
> |+--+|
> | Smith  |4 | 22 |
> | Miller |4 | 19 |
> | Wick   |4 | 18 |
> |+--+|
> #+TBLFM: @2$3..@>$3=remote(table1,@@#$7)

With this formula, your remote reference targets the field, that is in
the same row number as the computed field in final.

Here is the same table, with some extra columns that show the
intermediate steps:

   #+Name: final2
   | Name   | Some | ResSh1 | What is '@#'? | Which ref? |
   |+--++---+|
   | Smith  |4 | 22 | 2 | @2$7   |
   | Miller |4 | 19 | 3 | @3$7   |
   | Wick   |4 | 18 | 4 | @4$7   |
   |+--++---+|
   #+TBLFM: @2$3..@>$3=remote(table1,@@#$7)
   #+TBLFM: @2$4..@>$4=@#
   #+TBLFM: @2$5..@>$5='(concat "@" (format "%d" @#) "$" "7")

Bruno



>
>
>
> -- 
> Warning: Content may be disturbing to some audiences
> I strongly condemn Putin's war of aggression against Ukraine.
> I support to deliver weapons to Ukraine's military. 
> I support the NATO membership of Ukraine.
> I support the EU membership of Ukraine. 
> https://addons.thunderbird.net/en-US/thunderbird/addon/gmail-conversation-view/

Re: equivalent of VLOOKUP (in ods) to org-table

[Bruno Barbier]

Uwe Brauer  writes:

>>>> "BB" == Bruno Barbier  writes:
>> Did you check these lookup functions in the Org manual?
>>(info "(org) Lookup functions")
>
>
> Yes of course, but I am unable to obtain the same result as I do using
> the remote call.

Did you try something like this ?

   #+NAME: table1
   | Name   | Ex1 | Ex2 | Ex2 | Ex4 | Ex5 | ResSh1 |
   |+-+-+-+-+-+|
   | Smith  |   2 |   3 |   4 |   6 |   7 | 22 |
   | Miller |   2 |  10 |   1 |   1 |   5 | 19 |
   | Wick   |   1 |   2 |   3 |  10 |   2 | 18 |
   #+TBLFM: $7=vsum($2..$6);f2


   #+NAME: table2   
   | Name   | Ex1 | Ex2 | Ex2 | Ex4 | Ex5 | ResSh2 |
   |+-+-+-+-+-+|
   | Smith  |   8 |   3 |   5 |   8 |   9 | 33 |
   | Miller |   2 |   1 |   6 |   9 |   3 | 21 |
   | Wick   |   1 |   5 |   9 |   1 |   2 | 18 |
   #+TBLFM: $7=vsum($2..$6);f2

   | Name   |   | ResSh1 | ResSh2 | Total |
   |+---+++---|
   | Smith  | 4 | 22 | 33 |59 |
   | Miller | 4 | 19 | 21 |44 |
   | Wick   | 4 | 18 | 18 |40 |
   #+TBLFM: $3='(org-lookup-first $1 '(remote(table1, @I$1..@II$1)) 
'(remote(table1, @I$7..@II$7)))
   #+TBLFM: $5=vsum($2..$4)

IIUC, it seems to be the result that you're expecting.

Bruno


>
> Uwe 
> -- 
> Warning: Content may be disturbing to some audiences
> I strongly condemn Putin's war of aggression against Ukraine.
> I support to deliver weapons to Ukraine's military. 
> I support the NATO membership of Ukraine.
> I support the EU membership of Ukraine. 
> https://addons.thunderbird.net/en-US/thunderbird/addon/gmail-conversation-view/

Re: equivalent of VLOOKUP (in ods) to org-table

[Bruno Barbier]

Hi Uwe,

Uwe Brauer  writes:
> so the question is what is equivalent of VLOOKUP in org.

Did you check these lookup functions in the Org manual?

   (info "(org) Lookup functions")


Bruno.



>
> I came up with the remote command that results in a similar result, (I did 
> not want to use third party packages like 
> orgtbl-aggregate. 
>
> #+begin_src 
> #+Name: table1
> | Name   | Ex1 | Ex2 | Ex2 | Ex4 | Ex5 | ResSh1 |
> |+-+-+-+-+-+|
> | Smith  |   2 |   3 |   4 |   6 |   7 | 22 |
> | Miller |   2 |  10 |   1 |   1 |   5 | 19 |
> | Wick   |   1 |   2 |   3 |  10 |   2 | 18 |
> #+TBLFM: $7=vsum($2..$6);f2
>
>
> #+Name: table2   
> | Name   | Ex1 | Ex2 | Ex2 | Ex4 | Ex5 | ResSh2 |
> |+-+-+-+-+-+|
> | Smith  |   8 |   3 |   5 |   8 |   9 | 33 |
> | Miller |   9 |   4 |   6 |   9 |   3 | 31 |
> | Wick   |   1 |   5 |   9 |   1 |   2 | 18 |
> |+-+-+-+-+-+|
> #+TBLFM: $7=vsum($2..$6);f2
>
> #+Name: final
> | Name   | Some | ResSh1 | ResSh2 | Final |
> |+--+++---|
> | Smith  |4 | 22 | 33 |59 |
> | Miller |4 | 19 | 31 |54 |
> | Wick   |4 | 18 | 18 |40 |
> |+--+++---|
> #+TBLFM: $3=remote(table1,@@#$7)::$4=remote(table2,@@#$7)::$5=vsum($2..$4);f2
> #+end_src
>
>
> The syntax with @@#$7 look cryptic to me, but I found it somewhere and it 
> works
> if somebody can suggest a simpler command that is closer to VLOOKUP I would 
> be grateful
>
> Thanks and regards
>
> Uwe Brauer 
>
> -- 
> Warning: Content may be disturbing to some audiences
> I strongly condemn Putin's war of aggression against Ukraine.
> I support to deliver weapons to Ukraine's military. 
> I support the NATO membership of Ukraine.
> I support the EU membership of Ukraine. 
> https://addons.thunderbird.net/en-US/thunderbird/addon/gmail-conversation-view/

Re: Named columns in org tables [9.7-pre (release_9.6.9-797-g4d0f89]

[Bruno Barbier]

Hello Paul,

Paul Stansell  writes:

>  It seems to be on a todo list already as the following exists:
> - https://list.orgmode.org/877cqwbpa2@runbox.com

Right. Thanks for the link!

You just need to wait a little :-)

You should probably send an email to the emacs bug:

   bug#66213: named columns in org tables ()

that it may be closed.

Bruno

Re: Fwd: Named columns in org tables [9.7-pre (release_9.6.9-797-g4d0f89]

[Bruno Barbier]

Paul Stansell  writes:
>
> By the way, there is a small error in your example as your $3 should be $4.

Oops. Sorry. Thanks for reporting it.

Bruno

Re: Named columns in org tables [9.7-pre (release_9.6.9-797-g4d0f89]

[Bruno Barbier]

Hi Paul,

Paul Stansell  writes:

> Hello,
>
> On this page https://orgmode.org/manual/Advanced-features.html
> it says
> - '!' :: The fields in this line define names for the columns, so that
>   you may refer to a column as '$Tot' instead of '$6'.
> ...
> |---+++|
> | ! | c1 | c2 | c3 |
> | # |  1 |  2 ||
> | # |  3 |  4 ||
> |---+++|
> #+TBLFM: $c3 = $c1 + $c2
>
> Is this a bug?

It seems to me it's a bug in the documentation.

Currently, column names are not allowed in the LHS (Left Hand Side) of
the formula ('org-table-get-stored-formulas' assumes it).  Although I'm
not sure why.

In other words, you'll need to use the real column number
for the column that holds the results. This should work:

   | ! | c1 | c2 | c3 |
   |---+++|
   | # |  1 |  4 |  3 |
   | # |  3 | 10 |  7 |
   #+TBLFM: $3 = $c1 + $c2



Bruno

Re: Calc/TBLFM: how to conditionally insert hours:minutes?

[Bruno Barbier]

Hi Chris,

Chris Keschnat  writes:

>
> | 19:55:00 |
> | 00:00:40 |
> | 00:00:40 |
> #+TBLFM: $1=if(@# <= 1, 19:55, 40:01);T
>
> What would be the correct way to do this?
>

It seems that org tries to convert times back and forth when evaluating
formulas; but, in your case, it cannot convert them because it doesn't
spot them inside your formula.

You could switch to the "Lisp" syntax (as opposed to the "Calc" syntax),
and, write the conversion manually:

| 19:55:00 |
| 40:01:00 |
| 40:01:00 |
#+TBLFM: $1='(org-table-time-string-to-seconds (if (<= @# 1) "19:55" 
"40:01"));T

Note that, for Calc, 'X:Y' means the fraction 'X/Y' of the two
integers X and Y:

   40:01 = 40/1 = 40
   40:02 = 40/2 = 20
   40:03 = 40/3
   40:04 = 40/4 = 10


Bruno

Re: [PATCH] Add tests for ob-haskell (GHCi)

[Bruno Barbier]

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>> From 86a5443948fc84a6a412ccf49d0c537608f465a7 Mon Sep 17 00:00:00 2001
>> From: Bruno BARBIER 
>> Date: Fri, 18 Nov 2022 20:14:20 +0100
>> Subject: [PATCH 1/7] ob-haskell: Add tests for GHCi
>> ...
>
> Thanks!
> Applied, onto main.
> I also added a missing defvar declaration for
> `inferior-haskell-root-dir'.

Thanks Ihor!

Re: [PATCH] Add tests for ob-haskell (GHCi)

[Bruno Barbier]

Hi Ihor,

Sorry for the delay, thanks again for the ping.



Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>>>> +  (when (bufferp "*haskell*") (error "Conflicting buffer 
>>>> '*haskell*', rename it or kill it."))
>>>> +  (with-current-buffer session (rename-buffer "*haskell*")))
>>>
>>> So, you are now renaming the unique session buffer back to "*haskell*".
>>> And never rename it back to expected :session . Users might be 
>>> confused.
>>
>> I do rename it back once inf-haskell has initialized the buffer (after
>> run-haskell in the last version).
>
> A comment would help to clarify things for the readers.

Right. I've improved the comment. Thanks.


>>>> +(save-window-excursion
>>>> +  ;; We don't use `run-haskell' to not popup the buffer.
>>>> +  ;; And we protect default-directory.
>>>> +  (let ((default-directory default-directory))
>>>> +(inferior-haskell-start-process))
>>>




>> About 'default-directory', I'm not sure. Maybe the side effect is done
>> on purpose in inf-haskell.
>
> I can see the haskell-mode overrides default-directory with
> `inferior-haskell-root-dir', running ghci in that directory, if it is
> non-nil. Even with your let binding, it is calling for trouble when
> source block uses :dir header argument.
>
> Maybe we can bind `inferior-haskell-root-dir' to `default-directory'
> instead? `default-directory' is modified according to :dir by ob-core.el
> when necessary.

You may be right that we should use `inferior-haskell-root-dir' to
tell haskell-mode where to run the interpreter.  Done.



>> The function 'putStr' output the string without a newline on stdout
>> (as opposed to the function putStrLn that does add a newline).
>>
>> So, in GHCi, entering:
>>
>> putStr("4")
>> 
>> outputs "4" on stdout, then GHCi outputs the prompt, so we get:
>>
>> 4ghci> 
>>
>> In the end, 'org-babel-comint-with-output' gets this
>> 1ghci> 2ghci> 3ghci> 
>> ghci> org-babel-haskell-eoe
>> ghci> ghci>
>> 
>> and filters out everything as being GHCi prompts and the EOE.
>>
>> I'm not really expecting this to be fixed; I just wanted to record the
>> fact.
>
> We actually might be able to deal with this if we change the prompt and
> update comint-prompt-regexp to something more accurate.

I couldn't make it work by simply restricting the prompt.  I think
it's OK if character by character output doesn't work; ghci is about
line based interaction afterall.



>>>> Subject: [PATCH 11/13] lisp/ob-haskell.el: Fix how to use sessions
>>>>
>>>> +  (org-babel-haskell-with-session
>>>
>>> This kind of names are usually dedicated to macro calls. But
>>> `org-babel-haskell-with-session' is instead a function. I think a macro
>>> will be better. And you will be able to get rid of unnecessary lambda.
>>
>> That looks kind of complicated just to avoid one lambda in one call.
>> But, as I couldn't find a better name, I've translated it into a
>> macro.
>
> I think you misunderstood what I meant.
> See the attached diff on top of your patches that simplifies things a
> bit.
> diff --git a/lisp/ob-haskell.el b/lisp/ob-haskell.el
...
>  (defmacro org-babel-haskell-with-session (session-symbol params &rest body)
>"Get the session identified by PARAMS and run BODY with it.
.. 
> +  `(let* ((params ,params)
> +  (sn (cdr (assq :session params)))
> +  (session (org-babel-haskell-initiate-session sn params))
> +  (,session-symbol session)
> +  (one-shot (equal sn "none")))
> + (unwind-protect
> + (progn ,@body)

I don't think it's correct to create local variables like this in a
macro: we need to use uninterned symbols, else we may capture caller
variables (params, sn, session and one-shot).

I personnaly find it easier when I keep my macros as short as
possible, and, to do any non-trivial work in a function: easier to
read, to modify and to debug.




>>>> -(let ((buffer (org-babel-haskell-initiate-session session)))
>>>> +(let ((buffer (org-babel-haskell-initiate-session session params)))
>>>
>>> PARAMS argument is ignored by `org-babel-haskell-initiate-session'. I am
>>> not sure why you ar

Re: [PATCH] Add tests for ob-haskell (GHCi)

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

> A few months have passed since the last activity in this thread.
> Bruno, may I know if you are still interested to work on the patch?

Thanks for the reminder.  I'm definitely interested to closing that
thread.

I'll review my last changes and send my updated patch.

Thank you for your patience,

Bruno.

> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at .
> Support Org development at ,
> or support my work at 

Re: org-table, empty cells and nan

[Bruno Barbier]

Hi,

Uwe Brauer  writes:

> A solution is to put 0 in colum 4, but for some reasons which are a
> complicated to explain, I want to avoid that.
>
> | Name  | Exam1 | Exam2 | Exercises | Ex1_Ex2 | Total |
> |---+---+---+---+-+---|
> | Smith |   5.9 |  7.90 | 0 |5.81 |   5.8 |
> #+TBLFM: $6=if("$2" == "nan", string("NP"), 0.65*$2+0.25*$3+$4); E f-1
>
>
>
> Any idea how to deal with the situation?

Maybe this formula?

#+TBLFM: $6=if("$2" == "nan", string("NP"), 0.65*$2+0.25*$3+ if("$4" == 
"nan", 0, $4)); E f-1

Regards,

Bruno

Re: [PATCH] Add tests for ob-haskell (GHCi)

[Bruno Barbier]

Hi Ihor,

Thanks for the review.


Ihor Radchenko  writes:
> Bruno Barbier  writes:


> I can see that you limited the tests scope to :session blocks.
> Would it be possible to extend the existing tests to :compile yes case?
> From a glance, it does not look like you need to change much - Haskell
> behaviour should be similar with or without ghci.


Except for one line expressions, GHCi inputs and haskell modules are
two different things.  I doubt there will be much to share; that's why
I've focused from the start on GHCi only.

As I've a lot of other things that I'd like to do to improve my day to
day workflow, and as I'm barely using ob-haskell, I can't promise I'll
work on this any time soon.


>> From 9972b926f55cb970e0b520f8726a3684118017b6 Mon Sep 17 00:00:00 2001
>> From: Ihor Radchenko 
>> Date: Fri, 24 Mar 2023 11:20:22 +0100
>> Subject: [PATCH 02/13] org-babel-haskell-initiate-session: Remove secondary
>>  prompt
>>
>> * lisp/ob-haskell.el (org-babel-haskell-initiate-session): Set
>> secondary prompt to "".  If we do not do this, org-comint may treat
>> secondary prompts as a part of output.
>
>> +(sleep-for 0.25)
>> +;; Disable secondary prompt.
>
> It would be useful to explain the purpose of disabling the secondary
> prompt in the source code comment itself, not just in the commit
> message. It will improve readability.

Are you reviewing your own improvements ? :-)

Fixed. I've copied/pasted your explanation in the code.



>> From 352d18399961fedc45cc2d64007016426e1ecd40 Mon Sep 17 00:00:00 2001
>> From: Ihor Radchenko 
>> Date: Fri, 24 Mar 2023 11:26:00 +0100
>> Subject: [PATCH 04/13] * testing/lisp/test-ob-haskell-ghci.el: Enable fixed
>
> I do not see PATCH 03/13 in the attachments.

Sorry. I forgot it the first time. The email, after handling the
comments from Ruijie Yu, had it though.



>> From 7d66cff5cc23bb786cb2843f4326d2869512ccac Mon Sep 17 00:00:00 2001
>> From: Bruno BARBIER 
>> Date: Sat, 25 Mar 2023 10:06:44 +0100
>> Subject: [PATCH 06/13] ob-haskell: Implement sessions
>>
>> +  (unless session-name
>> +;; As haskell-mode is using the buffer name "*haskell*", we stay
>> +;; away from it.
>> +(setq session-name (generate-new-buffer-name "*ob-haskell*")))
>> +  (let ((session (get-buffer session-name)))
>
> session is not a buffer or nil, if no buffer named session-name exists.

The argument SESSION-NAME must be a string or nil (I added a test to
make clear that it must be a string).  Thus, session will either be nil or
a live buffer.



>> +(save-window-excursion
>> +  (or (org-babel-comint-buffer-livep session)
>
> Below, (org-babel-comint-buffer-livep session) is nil, which implies
> either that session is nil, does not exist, not live, or does not have a
> process attached.

ok. So, in our case, session is either nil, or it's a live buffer
without an attached process.

>
>> +  (let ((inferior-haskell-buffer session))
>> +(when (and (bufferp session) (not 
>> (org-babel-comint-buffer-livep session)))
>
> (not (org-babel-comint-buffer-livep session)) is always t here.

Right. I removed the test. Thanks.


> Also, session may be a killed buffer object. It is still a buffer, but
> not usable. See `buffer-live-p'.

By construction, if 'session' is a buffer, then, it is a live buffer.


>
>> +  (when (bufferp "*haskell*") (error "Conflicting buffer 
>> '*haskell*', rename it or kill it."))
>> +  (with-current-buffer session (rename-buffer "*haskell*")))
>
> So, you are now renaming the unique session buffer back to "*haskell*".
> And never rename it back to expected :session . Users might be 
> confused.

I do rename it back once inf-haskell has initialized the buffer (after
run-haskell in the last version).


>> +(save-window-excursion
>> +  ;; We don't use `run-haskell' to not popup the buffer.
>> +  ;; And we protect default-directory.
>> +  (let ((default-directory default-directory))
>> +(inferior-haskell-start-process))
>
> This is a workaround for a nasty side effect of running
> `inferior-haskell-start-process'. We should report this to haskell-mode
> developers, leaving appropriate comment in the code.

About 'run-haskell', I reverted my change: we're inside a
'save-window-excursion', which looks like the standard way to get rid
of unwanted popups (like ob-shell does).

About 'default-directory', I'm not sure. Ma

Re: Formulas on table cells containing '$'

[Bruno Barbier]

Jeff Trull  writes:

> While investigating an error executing a table formula I discovered that
> cells containing '$' cause column references to be executed even when no
> attempt is made to evaluate cell contents as code. Here's a simple example:
>

Confirmed.


org tries first to resolve all references, recursively.


   '(length '(@1$3..@I$3))

   { resolving @1$3..@I$3 into ("$200.00" "$1.13" "$301.22") }
  
=  '(length '("$200.00" "$1.13" "$301.22"))

   { resolving $200 into ...
  
ERROR: '$200' is an invalid reference.



It's probably not by designed, but, it's definitely a limitation of
the current implementation.


As a workaround, you could hide your reference in elisp and resolve it
manually using `org-table-get-range'. That way you can add/remove "$" as
needed.


For example:

 | 3/1/2023  | Deposit| $200.00 |
 | 3/13/2023 | Interest   | $1.13   |
 | 4/1/2023  | Deposit| $301.22 |
 |---++-|
 |   | Number of Transactions | 3   |
 |   | Total  | $502.35 |
 #+TBLFM: @4$3='(length (org-table-get-range (concat "@" "1$" "3..@" "I$" 
"3")))
 #+TBLFM: @5$3='(format "$%.2f" (apply '+ (mapcar (lambda (x) 
(string-to-number (substring x 1))) (org-table-get-range (concat "@" "1$" 
"3..@" "I$" "3")




Bruno

Re: [PATCH] Add tests for ob-haskell (GHCi)

[Bruno Barbier]

Ruijie Yu  writes:

> Minor remarks below regarding the patchset.
>
> Bruno Barbier  writes:
>
>> +;; Copyright (c) 2023  Free Software Foundation, Inc.
>
> lisp/org.el has only a single space, so probably single space here as well.

Done.

>> +
>> +;; Authors: Bruno BARBIER 
>> +
>> +;; This program is free software; you can redistribute it and/or modify
>> +;; it under the terms of the GNU General Public License as published by
>> +;; the Free Software Foundation, either version 3 of the License, or
>> +;; (at your option) any later version.
>
> Do we need the text for "part of GNU Emacs"?
>

I guess it doesn't harm: I added it, thanks.


>> +
>> +(defun test-ob-haskell-ghci--with-global-session-worker (todo)
>> +  "See `test-ob-haskell-ghci--with-global-session-worker'."
>
> This docstring doesn't say much and only refers to itself.  Maybe
> explain what it does?  (Or now that I look at it, potentially you wanted
> to refer to the macro `test-ob-haskell-ghci-with-global-session'
> instead.)

I've rewritten that function later ... which made the documentation even worse 
:-)
I've fixed it, thanks.


>
>> +(defun test-ob-haskell-ghci (args content &optional preamble unprotected)
>> +  "Execute the code block CONTENT in a new GHCi session; return the result.
>> +Add ARGS to the code block argument line.  Insert PREAMBLE
>> +before the code block.  When UNPROTECTED is non-nil, don't control
>> +which session is used (i.e. don't call
>> +`test-ob-haskell-ghci--with-global-session-worker')."
>> +  (when (listp content)
>> +(setq content (string-join content "\n")))
>> +  (unless preamble
>> +(setq preamble ""))
>> +  (let ((todo  (lambda ()
>
> One space.

AFAICS, the last version has only one space here.


>> +
>> + Not define  errors
>> +;;

> Single space?

It was an invisible 'd' actually; I repainted in black :-)
Thanks.


Thank you for your review,

Bruno

>From 136878a096eb9f459e97da6617f94ba84085db9b Mon Sep 17 00:00:00 2001
From: Bruno BARBIER 
Date: Fri, 18 Nov 2022 20:14:20 +0100
Subject: [PATCH 01/13] ob-haskell: Add tests for GHCi

testing/lisp/test-ob-haskell-ghci.el: New file.
---
 testing/lisp/test-ob-haskell-ghci.el | 454 +++
 1 file changed, 454 insertions(+)
 create mode 100644 testing/lisp/test-ob-haskell-ghci.el

diff --git a/testing/lisp/test-ob-haskell-ghci.el b/testing/lisp/test-ob-haskell-ghci.el
new file mode 100644
index 0..4023873de
--- /dev/null
+++ b/testing/lisp/test-ob-haskell-ghci.el
@@ -0,0 +1,454 @@
+;;; test-ob-haskell-ghci.el --- tests for ob-haskell.el GHCi  -*- lexical-binding: t; -*-
+
+;; Copyright (c) 2023 Free Software Foundation, Inc.
+;; Authors: Bruno BARBIER 
+
+;; This file is part of GNU Emacs.
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+;;
+
+ Useful references
+;;
+;;  - https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html
+;;  - GHCi manual: https://downloads.haskell.org/ghc/latest/docs/users_guide/ghci.html
+ FIXME: Random failures
+;;
+;; To increase the chances of failure when running tests, you can use this command line:
+;;
+;;(for I in 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10; do make 'BTEST_OB_LANGUAGES=haskell' BTEST_RE='haskell' test-dirty & done) 2>&1 | grep FAILED
+;;
+
+ Status
+;;
+;; All the tests should succeed (except for random failures); those
+;; flagged with ":expected-result :failed" are known
+;; limitations/bugs.  Tested with (2023-03-18):
+;;
+;; | emacs-version |  29.0.60 |
+;; | org-version   | main@4cad6c8ea (Mar 16 2023) |
+;; | haskell-mode  | master@20d4e23 (Mar 4  2023) |
+;; | ghci  |9.0.2 |
+
+
+;;; Code:
+;;
+
+(require 'org-test "../testing/org-test")
+(org-test-for-executable "ghci")
+(unless (featurep 'haskell-mode)
+  (signal 'missing-test-dependency "haskell-mode"))
+
+
+;;; Helpers
+;;
+
+(defun test-ob-haskell-ghci--with-glob

Re: [PATCH] Add tests for ob-haskell (GHCi)

[Bruno Barbier]

Hi Ihor,

Sorry for the delay.

Bruno Barbier  writes:

> Ihor Radchenko  writes:
>
>> Bruno Barbier  writes:
>>
>>> Note that I've changed the tests about errors; I'm now expecting
>>> ob-haskell to raise errors. I'm not sure what we should expect to be
>>> consistent with other org babel backends.
>>
>> Errors are usually displayed separately, using
>> `org-babel-eval-error-notify'.
>
> I'll see what can be done with GHCi and use this if possible.

Unfortunately, no progress on this.  And I don't really have time to
work on this.



>>> +;; To increase the chances of failure when running tests, you can use this 
>>> command line:
>>> +;;
..
>>> +;; | ghci  |9.0.2 |
>>
>> You can probably remove this.
>
> Definitely. I'll do. Thanks.

Done.



>>> +  (when (and session-name (string= session-name "none"))
>>> +(setq session-name nil))
>>> +  (unless session-name
>>> +;; As haskell-mode is using the buffer name "*haskell*", we stay
>>> +;; away from it.
>>> +(setq session-name (generate-new-buffer-name "*ob-haskell*")))
>>
>> This will make ob-haskell spawn a separate ghci process buffer every
>> single time a user runs non-session src block. And the buffer is not
>> closed after getting the result.

> Very good point!
>
> I will update this to use the same buffer named "*ob-haskell*" when the
> user doesn't set the session name.  I guess it's consistent with other
> org-babel backends.

I've changed the way session works. 'ob-haskell' should now destroy
temporary sessions.  Thanks again for spotting that mistake.


Let me know if you see further improvement before pushing this.

Thanks,

Bruno


>From 9ef867cd2cf89e77b5c5a5a7090fd37b1702e06a Mon Sep 17 00:00:00 2001
From: Bruno BARBIER 
Date: Fri, 18 Nov 2022 20:14:20 +0100
Subject: [PATCH 01/13] ob-haskell: Add tests for GHCi

testing/lisp/test-ob-haskell-ghci.el: New file.
---
 testing/lisp/test-ob-haskell-ghci.el | 453 +++
 1 file changed, 453 insertions(+)
 create mode 100644 testing/lisp/test-ob-haskell-ghci.el

diff --git a/testing/lisp/test-ob-haskell-ghci.el b/testing/lisp/test-ob-haskell-ghci.el
new file mode 100644
index 0..aba94d73f
--- /dev/null
+++ b/testing/lisp/test-ob-haskell-ghci.el
@@ -0,0 +1,453 @@
+;;; test-ob-haskell-ghci.el --- tests for ob-haskell.el GHCi  -*- lexical-binding: t; -*-
+
+;; Copyright (c) 2023  Free Software Foundation, Inc.
+
+;; Authors: Bruno BARBIER 
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+;;
+
+ Useful references
+;;
+;;  - https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html
+;;  - GHCi manual: https://downloads.haskell.org/ghc/latest/docs/users_guide/ghci.html
+ FIXME: Random failures
+;;
+;; To increase the chances of failure when running tests, you can use this command line:
+;;
+;;(for I in 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10; do make 'BTEST_OB_LANGUAGES=haskell' BTEST_RE='haskell' test-dirty & done) 2>&1 | grep FAILED
+;;
+
+ Status
+;;
+;; All the tests should succeed (except for random failures); those
+;; flagged with ":expected-result :failed" are known
+;; limitations/bugs.  Tested with (2023-03-18):
+;;
+;; | emacs-version |  29.0.60 |
+;; | org-version   | main@4cad6c8ea (Mar 16 2023) |
+;; | haskell-mode  | master@20d4e23 (Mar 4  2023) |
+;; | ghci  |9.0.2 |
+
+
+;;; Code:
+;;
+
+(require 'org-test "../testing/org-test")
+(org-test-for-executable "ghci")
+(unless (featurep 'haskell-mode)
+  (signal 'missing-test-dependency "haskell-mode"))
+
+
+;;; Helpers
+;;
+
+(defun test-ob-haskell-ghci--with-global-session-worker (todo)
+  "See `test-ob-haskell-ghci--with-global-session-worker'."
+  (when (get-buffer "*haskell*")
+(error "A buffer named '*haskell*' exists.  Can't safely test haskell blocks"))
+ 

Re: [PATCH] Add tests for ob-haskell (GHCi)

[Bruno Barbier]

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>> Note that I've changed the tests about errors; I'm now expecting
>> ob-haskell to raise errors. I'm not sure what we should expect to be
>> consistent with other org babel backends.
>
> Errors are usually displayed separately, using
> `org-babel-eval-error-notify'.

I'll see what can be done with GHCi and use this if possible.
Thanks.


>> +;; To increase the chances of failure when running tests, you can use this 
>> command line:
>> +;;
>> +;;(for I in 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 
>> 7 8 9 10; do make 'BTEST_OB_LANGUAGES=haskell' BTEST_RE='haskell' test-dirty 
>> & done) 2>&1 | grep FAILED
>> +;;
>> +
>> + Status
>> +;;
>> +;; All the tests should succeed (except for random failures); those
>> +;; flagged with ":expected-result :failed" are known
>> +;; limitations/bugs.  Tested with (2023-03-18):
>> +;;
>> +;; | emacs-version |  29.0.60 |
>> +;; | org-version   | main@4cad6c8ea (Mar 16 2023) |
>> +;; | haskell-mode  | master@20d4e23 (Mar 4  2023) |
>> +;; | ghci  |9.0.2 |
>
> You can probably remove this.

Definitely. I'll do. Thanks.



>> +(`value (org-babel-comint-with-output
>> +(session org-babel-haskell-eoe nil full-body)
>> +  (insert "__LAST_VALUE_IMPROBABLE_NAME__=()::()\n")
>> +  (comint-send-input nil t)
>> +  (insert full-body)
>> +  (comint-send-input nil t)
>> +  (insert "__LAST_VALUE_IMPROBABLE_NAME__=it\n")
>> +  (comint-send-input nil t)
>> +  (insert (concat "putStrLn (\"\\\"\" ++ " 
>> org-babel-haskell-eoe " ++ \"\\\"\")\n"))
>
> Why not simply putStrLn ("\"" ++ show it ++ "\"") ?
>

I'm not sure I understand. I'm using the first
'org-babel-comint-with-output' to execute the source block and save the
last value (the "it" variable). Then, I'm using a second
'org-babel-comint-with-output' to make sure the output of this one
contains only the last value. If I display "it" in the first block, I
will not be able to differentiate between some output, previous values
and the last value. And, printing "org-babel-haskell-eoe" updates the
"it" variable (the last value becoming "()"), so I have to store the
real "it" somewhere.

Am I missing something ?

>> +  (when (and session-name (string= session-name "none"))
>> +(setq session-name nil))
>> +  (unless session-name
>> +;; As haskell-mode is using the buffer name "*haskell*", we stay
>> +;; away from it.
>> +(setq session-name (generate-new-buffer-name "*ob-haskell*")))
>
> This will make ob-haskell spawn a separate ghci process buffer every
> single time a user runs non-session src block. And the buffer is not
> closed after getting the result.

Very good point!

I will update this to use the same buffer named "*ob-haskell*" when the
user doesn't set the session name.  I guess it's consistent with other
org-babel backends.



Thank you very much for the review and you help!

Bruno

>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] Add tests for ob-haskell (GHCi)

[Bruno Barbier]

Ihor Radchenko  writes:

> I investigated further and now applied a set of patches that improves
> prompt filtering in org-comint.
> ..

Thanks Ihor.

I don't get random failures anymore.  And thanks for fixing the tests
that were incorrect.

Thanks Pareto for mentioning the possibiliy to customize GHCi prompts.



> The remaining tests fall into two categories:
> 1. Tests trying to test :results value vs. :results output in sessions.
> 2. Tests trying to test for multiple sessions not interfering each
>other.

About :results output vs :results value, I decided to use the "it"
variable, i.e. to ask GHCi what was the last value (see [1]).

About sessions, I decided to rename the buffer as you suggested. That's
a partial workaround that should work well enough, until haskell-mode
provides a way to choose the buffer name.

I've updated the tests and the expected results.

Note that I've changed the tests about errors; I'm now expecting
ob-haskell to raise errors. I'm not sure what we should expect to be
consistent with other org babel backends.

I also did some modifications in my initial patch in-place.

Bruno


[1]:https://downloads.haskell.org/ghc/latest/docs/users_guide/ghci.html#the-it-variable

>From 46e8fa78574908a15fe6eb82a2cca5d2f537c78e Mon Sep 17 00:00:00 2001
From: Bruno BARBIER 
Date: Fri, 18 Nov 2022 20:14:20 +0100
Subject: [PATCH 1/7] ob-haskell: Add tests for GHCi

testing/lisp/test-ob-haskell-ghci.el: New file.
---
 testing/lisp/test-ob-haskell-ghci.el | 453 +++
 1 file changed, 453 insertions(+)
 create mode 100644 testing/lisp/test-ob-haskell-ghci.el

diff --git a/testing/lisp/test-ob-haskell-ghci.el b/testing/lisp/test-ob-haskell-ghci.el
new file mode 100644
index 0..aba94d73f
--- /dev/null
+++ b/testing/lisp/test-ob-haskell-ghci.el
@@ -0,0 +1,453 @@
+;;; test-ob-haskell-ghci.el --- tests for ob-haskell.el GHCi  -*- lexical-binding: t; -*-
+
+;; Copyright (c) 2023  Free Software Foundation, Inc.
+
+;; Authors: Bruno BARBIER 
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+;;
+
+ Useful references
+;;
+;;  - https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html
+;;  - GHCi manual: https://downloads.haskell.org/ghc/latest/docs/users_guide/ghci.html
+ FIXME: Random failures
+;;
+;; To increase the chances of failure when running tests, you can use this command line:
+;;
+;;(for I in 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10; do make 'BTEST_OB_LANGUAGES=haskell' BTEST_RE='haskell' test-dirty & done) 2>&1 | grep FAILED
+;;
+
+ Status
+;;
+;; All the tests should succeed (except for random failures); those
+;; flagged with ":expected-result :failed" are known
+;; limitations/bugs.  Tested with (2023-03-18):
+;;
+;; | emacs-version |  29.0.60 |
+;; | org-version   | main@4cad6c8ea (Mar 16 2023) |
+;; | haskell-mode  | master@20d4e23 (Mar 4  2023) |
+;; | ghci  |9.0.2 |
+
+
+;;; Code:
+;;
+
+(require 'org-test "../testing/org-test")
+(org-test-for-executable "ghci")
+(unless (featurep 'haskell-mode)
+  (signal 'missing-test-dependency "haskell-mode"))
+
+
+;;; Helpers
+;;
+
+(defun test-ob-haskell-ghci--with-global-session-worker (todo)
+  "See `test-ob-haskell-ghci--with-global-session-worker'."
+  (when (get-buffer "*haskell*")
+(error "A buffer named '*haskell*' exists.  Can't safely test haskell blocks"))
+  (unwind-protect (funcall todo)
+;; Kill the "*haskell*" buffer to not pollute other tests.
+(when-let ((hb (get-buffer "*haskell*")))
+  (with-current-buffer hb
+(let ((kill-buffer-query-functions nil)
+  (kill-buffer-hook nil))
+  (kill-buffer hb))
+
+(defmacro test-ob-haskell-ghci-with-global-session (&rest body)
+  "Eval BODY in a new session, then destroy the session.
+The library ob-haskell doesn't implement session yet.  It will
+always use a buffer named \"*haskell*\".  We kill that buffer
+after the source block execution.  To be safe, we fail if such a
+buffer already exists."
+  `(test-ob-haskell-ghc

Re: [PATCH] Add tests for ob-haskell (GHCi)

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

> From a first look, random failures appear to be related to session
> initialization.

My function 'test-ob-haskell-ghci' should protect against that; it
ensures the "*haskell*" buffer is always new.  From what I understand,
this is an input buffering problem: inputs are not full lines.

> It appears that ob-haskell is re-using sessions even
> if :session is nil (default). And multiple session not allowed?? (I am
> looking at `org-babel-haskell-initiate-session', which ignores session
> parameter completely).

>From what I understood, the buffer "*haskell*" is coming from
haskell-mode. And, to make it work might require some changes there too.

Thanks for having a look,
Let me know,


Bruno


>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at .
> Support Org development at ,
> or support my work at 

Re: Haskell code blocks

[Bruno Barbier]

Hi Ihor,

>> Bruno Barbier  writes:
>>
>>> Sorry, I'm still working on adding a 'test-ob-haskell.el', when I have
>>> some spare time, but I'm unable so far to find tests that I can't
>>> reliably break.

FTR: I've opened a new thread, with a patch containing tests:

   https://lists.gnu.org/archive/html/emacs-orgmode/2023-03/msg00274.html

Bruno.

[PATCH] Add tests for ob-haskell (GHCi)

[Bruno Barbier]

Hi all,

I've collected some tests about ob-haskell, using GHCi.

This patch is not ready for final review.  It's a preliminary version.

These tests are about GHCi, and only about GHCi (that's why I named
the file 'test-ob-haskell-ghci.el').


Some tests are revealing bugs; they are flagged with:

:expected-result :failed


All the tests will randomly fail; that is an ob-haskell bug (see
the command 'test-ob-haskell-ghci.el' to make them fail).


Some tests might be questionable, revealing more design choices and/or
backend capabilities than the expected behavior.


Solving the random failure bug should be a priority, in my opinion, as
it makes the testing process unreliable.  Adding support for sessions
would also be a good thing, so that one test doesn't contaminate others.


FTR, here are the tools that I have used:

| emacs-version |  29.0.60 |
| org-version   | main@4cad6c8ea (Mar 16 2023) |
| haskell-mode  | master@20d4e23 (Mar 4  2023) |
| ghci  |9.0.2 |



Bruno

>From c085fac2fcb429f7e643df8e4fff3a4ae1665d07 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER 
Date: Fri, 18 Nov 2022 20:14:20 +0100
Subject: [PATCH] ob-haskell: Add tests for GHCi

testing/lisp/test-ob-haskell-ghci.el: New file.
---
 testing/lisp/test-ob-haskell-ghci.el | 428 +++
 1 file changed, 428 insertions(+)
 create mode 100644 testing/lisp/test-ob-haskell-ghci.el

diff --git a/testing/lisp/test-ob-haskell-ghci.el b/testing/lisp/test-ob-haskell-ghci.el
new file mode 100644
index 0..4b1e4669b
--- /dev/null
+++ b/testing/lisp/test-ob-haskell-ghci.el
@@ -0,0 +1,428 @@
+;;; test-ob-haskell-ghci.el --- tests for ob-haskell.el GHCi  -*- lexical-binding: t; -*-
+
+;; Copyright (c) 2023  Free Software Foundation, Inc.
+
+;; Authors: Bruno BARBIER 
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+;;
+
+ Useful references
+;;
+;;  - https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html
+
+ FIXME: Random failures
+;;
+;; To increase the chances of failure when running tests, you can use this command line:
+;;
+;;(for I in 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10; do make 'BTEST_OB_LANGUAGES=haskell' BTEST_RE='haskell' test-dirty & done) 2>&1 | grep FAILED
+;;
+
+ Status
+;;
+;; All the tests should succeed (except for random failures); those
+;; flagged with ":expected-result :failed" are known
+;; limitations/bugs.  Tested with (2023-03-18):
+;;
+;; | emacs-version |  29.0.60 |
+;; | org-version   | main@4cad6c8ea (Mar 16 2023) |
+;; | haskell-mode  | master@20d4e23 (Mar 4  2023) |
+;; | ghci  |9.0.2 |
+
+
+;;; Code:
+;;
+
+(require 'org-test "../testing/org-test")
+(org-test-for-executable "ghci")
+(unless (featurep 'haskell-mode)
+  (signal 'missing-test-dependency "haskell-mode"))
+
+
+;;; Helpers
+;;
+
+(defun test-ob-haskell-ghci--with-global-session-worker (todo)
+  "See `test-ob-haskell-ghci--with-global-session-worker'."
+  (when (get-buffer "*haskell*")
+(error "A buffer named '*haskell*' exists.  Can't safely test haskell blocks"))
+  (unwind-protect (funcall todo)
+;; Kill the "*haskell*" buffer to not pollute other tests.
+(when-let ((hb (get-buffer "*haskell*")))
+  (with-current-buffer hb
+(let ((kill-buffer-query-functions nil)
+  (kill-buffer-hook nil))
+  (kill-buffer hb))
+
+(defmacro test-ob-haskell-ghci-with-global-session (&rest body)
+  "Eval BODY in a new session, then destroy the session.
+The library ob-haskell doesn't implement session yet.  It will
+always use a buffer named \"*haskell*\".  We kill that buffer
+after the source block execution.  To be safe, we fail if such a
+buffer already exists."
+  `(test-ob-haskell-ghci--with-global-session-worker (lambda () ,@body)))
+
+(defun test-ob-haskell-ghci (args content &optional preamble unprotected)
+  "Execute the code block CONTENT in a new GHCi session; return the result.
+Add ARGS to the code block argument line.  Insert PREAMBLE
+be

Re: Haskell code blocks

[Bruno Barbier]

Hi Ihor,

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>> Sorry, I'm still working on adding a 'test-ob-haskell.el', when I have
>> some spare time, but I'm unable so far to find tests that I can't
>> reliably break.
>
> May I know if you are still working on this?
> Do you need any help?

My tests were too easy to break. I tried to learn more about
haskell-mode, but I got lost. Then, it got down in my todo list.

I'll check what I have this week-end and come back to you.  But, that
might just be a set of tests that randomly fail.  Anyway, we'll decide
from there.

Thanks for your ping.

Bruno





>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: org-babel guile source block bug in handling multiple values

[Bruno Barbier]

Hi Zelphir,

Zelphir Kaltstahl  writes:

> On 3/7/23 20:52, Bruno Barbier wrote:

> Also thanks for the idea with sessions + separate import source block. I 
> thought 
> that should work, but apparently that also has the same error, when running 
> for 
> the first time:
>
> ...

Oh, I see. I tested something way simpler :-)

First, one block to open and configure the guile session.

 #+begin_src scheme :session "!guile" :results silent
 (import (except (rnrs base) error vector-map)
  (only (guile)
lambda*
λ)
  ;; let-values
  (srfi srfi-11))
 #+end_src

Then, you can get to work and evaluate as many blocks as you like in
that session:

 #+begin_src scheme :session "!guile" :results output replace drawer :var 
x=1 :var y=2
 (let-values ([(a b) (values x y)])
(simple-format #t "~a ~a\n" a b))
 #+end_src

 #+RESULTS:
 :results:
 1 2
 :end:

Bruno

Re: visual-line-mode don't play well with org-latex-preview

[Bruno Barbier]

Hi Chris,

chris  writes:

> On Thursday, 2 March 2023 20:00:14 CET Bruno Barbier wrote: 
>> The behavior looks the same using only text-mode (without using org).
>> You should probably report this as an Emacs bug (see M-x
>> report-emacs-bug).
>
> What command for latex preview in text mode? (I'll probably figure that out.)
>

org-latex-preview. Why ? :-)


> Yes, I should definitely report that bug.

In case it helps you to report the bug, here is the minimal way I found
to reproduce it (no latex, no org):

 #+begin_src elisp
   (let ((b (generate-new-buffer "=bug")))
 (with-current-buffer b
   (dotimes (_ 110) (insert "Hello "))
   (insert-image (svg-image (let ((s (svg-create 250 9)))
   (svg-line s 0 0 250 9 :stroke-color "green")
   s)))
   (org-mode)
   (visual-line-mode 1)
   (pop-to-buffer b)))
 #+end_src

Bruno

Re: org-babel guile source block bug in handling multiple values

[Bruno Barbier]

Zelphir Kaltstahl  writes:


> If org merely wraps in a `let`, it should not notice any of the multiple 
> values 
> business, because that is something done internally in `let-values`.
>

The "let", to define the org variables, ends up putting the "import"
inside the scheme expression, like this:

  ;; -*- geiser-scheme-implementation: guile -*-
  (let ((x '1)
(y '2))
  (import (except (rnrs base) error vector-map)
   (only (guile)
 lambda*
 λ)
   ;; let-values
   (srfi srfi-11))

  (let-values ([(a b) (values x y)])
 (simple-format #t "~a ~a\n" a b))
  )


which raises an error when evaluated the first time (the second time,
the "import" has already imported the new "let-values" feature
(srfi-11), so, the evaluation works). You can test this manually in a
guile session.

I guess you'll have to use sessions, and do the "import" in a separate
block first.

Bruno

Re: visual-line-mode don't play well with org-latex-preview

[Bruno Barbier]

Hi Chris,

chris  writes:

> Hi all,
>
> ```
> hello hello hello hello hello hello hello hello hello hello hello hello hello 
> hello hello  \(\text {hello hello hello hello hello hello hello hello}\)
> ```
>
> If you put the above text in a org file, and activate `visual-line-mode` and 
> `org-latex-preview`, depending on the length of the initial part of the text, 
> the generated png/svg image from latex expression can be displayed in full, 
> partially hidden or totally hidden.
>

I'm observing the same behavior.

The preview is not always fully visible.  But, after adding some text
(at least one space after the preview), Emacs wraps as needed, so that
the preview remains visible.  Emacs is just not wrapping correctly in
that edge case, where there is nothing at all after the latex preview.

The behavior looks the same using only text-mode (without using org).
You should probably report this as an Emacs bug (see M-x
report-emacs-bug).

Bruno

Re: ox-rst still working?

[Bruno Barbier]

Angel de Vicente  writes:

> Hello,
>
> I was trying to export an .org file to .rst. I have ox-rst 20200815.1511
> installed, and I have run (require 'ox-rst), but despite this, there is
> no option for rsT in the Org Export Dispatcher. Did I miss something to
> make it work?
>

Maybe this know issue, opened in 2017?
   https://github.com/msnoigrs/ox-rst/issues/35

Bruno

Re: RFC on implementation adding buttons beside headings

[Bruno Barbier]

pareto optimal  writes:

> I found the only way to place the button where I wanted was to insert
> some blank space after the heading. Is that expected and best practice
> or is there some other way to do it?

FWIW, instead of inserting some blank spaces, you could overlay existing
characters, you'll just need to re-inject them at display time so that
the display doesn't change. That way, you don't modified the text.


Here is a way to add a button, in place, without modifying the
buffer.  The trick is to overlay the EOL and to re-inject an EOL
using the "after-string" property.


 (with-current-buffer (generate-new-buffer "=test")
   (cl-flet ((run (f) (lambda (&rest args) (interactive) (funcall f
 (let (pos)
   (insert "* My title\n")
   (setq pos (1- (point)))
   (insert "An amazing note.\n\n")
   (insert "** My 2 title\n")
   (insert "An even more amazing note.\n")
   (org-mode)
   (make-button pos (1+ pos)
'before-string "\n\n"
'display '((height 2) "Click me!")
'after-string "\n\n"
'face 'custom-button
'action (run (lambda () (message "Thanks!")))
)))
   (pop-to-buffer (current-buffer))
   )
   
Note that the above expression requires lexical-binding.


Bruno

Re: Unicode problem with export of literal contents

[Bruno Barbier]

Jens Lechtenboerger  writes:

> On 2023-02-20, Bruno Barbier wrote:
>
> However, if I use insert-file-contents-literally with a unicode
> file, I do *not* have to set the coding-system-for-write.  This just
> works:
>
>(with-temp-buffer
>   (insert-file-contents-literally "~/unicode.org")
>   (secure-hash 'md5 (current-buffer)))

Humm. Emacs is amazing: it managed to guess the right encoding, from the
buffer context, probably...

But, what you are giving to 'org-export-string-as' is not the buffer,
it's a string. So, let's try the same without using an org function:

 (with-temp-buffer
   (insert (with-temp-buffer
 (insert-file-contents-literally "~/unicode.org")
 (buffer-string)))
   (secure-hash 'md5 (current-buffer)))
   
And, that fails, requesting an encoding.


> In the context of Org export, secure-hash seems to require a coding
> system.  Why?

I'm not an expert, so, you'll need to confirm with other sources.  But
secure-hash requires an encoding in all cases, to compute the hash of
some text, because it needs the array of bytes that represents that text
to compute its hash.

I don't see any bug in org, and, I don't see any bug in secure-hash either.

You literally shoud stop using "literally" ;-)

And, you might want to read:
   (info "(elisp) Non-ASCII Characters")



Bruno


>
> Best wishes
> Jens

Re: Unicode problem with export of literal contents

[Bruno Barbier]

Jens Lechtenboerger  writes:

> On 2023-02-17, Bruno Barbier wrote:
>
>> Here is a way to reproduce that doesn't use org, in case it might help
>> to manully fix your encoding issue:
>>
>>(with-temp-buffer
>>   (insert "Lechtenb\303\266rger")
>>   (let ((buffer-file-name (make-temp-file "mailtest")))
>> (save-buffer)))
>>
>> Does it work with your old config (with your old org) ?
>
> This also asks for an encoding.

If you're always using utf-8, here is a way to force it so that
secure-hash can compute the hash. This should work:

   (with-temp-buffer
  (let ((coding-system-for-write 'utf-8))
(insert "Lechtenb\303\266rger")
(secure-hash 'md5 (current-buffer

Without setting coding-system-for-write to utf-8, it asks for an
encoding:

   (with-temp-buffer
  (insert "Lechtenb\303\266rger")
  (secure-hash 'md5 (current-buffer)))


I'm still no getting your use case, but, let's hope that this naive hack
is enough for you :-)


Bruno


> Best wishes
> Jens

Re: Unicode problem with export of literal contents

[Bruno Barbier]

Jens Lechtenboerger  writes:

> So, maybe my question is: Must text be decoded for Org mode from now on?

Yes. Since forever.  Emacs must know how to read/write from/to files and
what text to display to you. Org is just relying on Emacs for that part.

Bruno

Re: Unicode problem with export of literal contents

[Bruno Barbier]

Jens Lechtenboerger  writes:

> On 2023-02-17, Ihor Radchenko wrote:
>
>> Jens Lechtenboerger  writes:
>>
>
>> Not a bug. You need to fix your files with improper encoding.
>
> The file has the proper encoding.  I insert literally on purpose as
> stated above.

IIUC, the file has the proper encoding. But, when loading it with
`insert-file-contents-literally', it doesn't: that's part of the
"literally" feature I guess.

When loading it with `insert-file-contents', it should work (it does in
my case).


Here is a way to reproduce that doesn't use org, in case it might help
to manully fix your encoding issue:

   (with-temp-buffer
  (insert "Lechtenb\303\266rger")
  (let ((buffer-file-name (make-temp-file "mailtest")))
(save-buffer)))

Does it work with your old config (with your old org) ?

What kind of failure do you get elsewhere if you let Emacs use the
correct encoding (i.e. if you use `insert-file-contents') ?



Bruno

Re: Unicode problem with export of literal contents

[Bruno Barbier]

Hi Jens,

Jens Lechtenboerger  writes:

> ...
> Note that I insert contents literally because I do not want
> ‘find-file-hook’, automatic uncompression, etc. (which are avoided
> according to the doc string of insert-file-contents-literally).
>
> Could the old behavior be restored?

By using `insert-file-contents-literally' (as opposed to
`insert-file-contents'), you're also forbidding Emacs to decode the
binary content of your file into text.

My guess is that it was working by chance in previous versions.

In case somebody might help you, here is a simple way to trigger the
encoding question with a recent version of org (mine is Org mode version 9.6.1).

   (with-temp-buffer
  (insert "Lechtenb\303\266rger")
  (org-mode))



Bruno

Re: Problem with let/cl-letf binding stuff with org-capture

[Bruno Barbier]

Arthur Miller  writes:

>
> Anyway, a follow question: Where is the source code?!

lisp/textmodes/string-edit.el

> Normally the help window says "... is a Lisp function in ."
>

My Emacs tells me it's an autoloaded function with the usual link.

Bruno

Re: [BUG] Incorrect display of folded headings after cycling with subheading with VISIBILITY content/children

[Bruno Barbier]

Daniel Hubmann  writes:

> ...
> After using org-cycle-overview (or org-cycle-content) and then using
> org-cycle-set-visibility-according-to-property I get this:
>
> * Heading Lvl 1...** Heading Lvl 2 with VISIBILITY content...
> *** Heading Lvl 3...
>

I'm observing the same behavior.

Org mode version 9.6.1

Bruno

Re: Problem with let/cl-letf binding stuff with org-capture

[Bruno Barbier]

Ihor Radchenko  writes:

> Note that Emacs 29 has `read-string-from-buffer'.

I completely missed the fact that it was already in Emacs.

I'll definitely use the string-edit version (the version
with callbacks).

Thanks Ihor.

Bruno

Re: Problem with let/cl-letf binding stuff with org-capture

[Bruno Barbier]

Arthur Miller  writes:

> Bruno Barbier  writes:
>
>>
>
> That looks very nice indeed. I am not aware of that package, I will definitely
> use it somewhere, sometime. But with that we are getting now into 1K extra 
> sloc
> solution. With this experiment, I was mostly interesting to see how I can 
> re-use
> what already is in Emacs. Lisps are great for prototyping and writing new
> software. Legend says that Steele was cranking out 10 intepreters a week at 
> his
> time :). I don't know how true it is, just read it in some blog, but the
> point is that we are rather typing fast new pieces instead of learning how
> to re-use stuff, which in the long run I believe is a bad habit, since we are
> debugging bits that does the same what someone else already debugged. Don't 
> get
> me wrong, I have nothing against using someones package, certainly not
> with-editor, seems like a good package; I am just talking about programming in
> general.

... And the piece of code, that I just wrote and sent, was already
available in Emacs (see Ihor's email). I've definitely failed to do my
research before coding ... I definitely get your point! :-)

> This was also me trying to get better in Emacs Lisp, but I have to admit I am 
> a
> bit dissapointed we can't just override function bindings with cl-flet. 
> Function
> slot is just another slot :).
>
> I am using cl-labels and cl-flet myself to introduce "local" functions, to not
> pollute global namespace (symbol table), which they are good for, and to keep
> code local where it is used, and I think they are also slightly more clear 
> then
> lambdas. They are good for that, unfortunately it is a bit of dissapoitment 
> that
> we can't locally bind functions slots other then actually overriding them 
> globally.

Since I've learned that cl-flet and cl-labels are local lexically, I'm
using them more and more: that removes lot of noisy funcalls, compare to
lambdas.


> And for the last time: I am not using this version of read-string, I don't 
> need
> it myself; it was just me thinking how to implement something after reading a
> blog post. Anyway, thanks for the input, it was valuable to me.

Got it. Thanks for your patience and your valuable inputs too.

And I'm definitely going to use the version that is in Emacs. Thanks for
this too.

best,

Bruno

Re: Problem with let/cl-letf binding stuff with org-capture

[Bruno Barbier]

Hi Arthur,

Arthur Miller  writes:

> Bruno Barbier  writes:
>
> ...  but I feel a
> bit of passive aggressivity here, for no good reason tbh.

I'm just trying to help, giving some valid or invalid advices.  I'm
sorry that what I wrote, and how I wrote it, made you feel that way.

>>
>> Yes, let binding is fundamental. But I think it's the first time I see
>> 'cl-letf' with the 'symbol-function' place.
>
> https://nullprogram.com/blog/2017/10/27/
> https://endlessparentheses.com/understanding-letf-and-how-it-replaces-flet.html
> https://stackoverflow.com/questions/39550578/in-emacs-what-is-the-difference-between-cl-flet-and-cl-letf
>

Thanks for these links. I like cl-flet and cl-labels :-)


>>> but I am not sure if I can do anything here without introducing at-least an
>>> extra keymap, to not install into the org-capture-mode-map, so I can as well
>>> create a minor mode, but at this point it is not much different than
>>> re-invinting the read-string, so I'll terminate my experiment here :).
>>
>> You can replace the buffer keymap with a keymap that only contain your custom
>> keys, and inherits everything else from org-capture-mode-map.
>
> Isn't that what I wrote: introducing an extra keymap?
> Of course I can solve the problem differently, but that was not what question
> was about :).

Right. Even when inheriting from the old keymap, it's still building a
new keymap.  Sorry :-)


> Well, I definitely understand you, and agree that overwriting function for
> everyone and everything is not the best idea, but unfortunately bindings work 
> as
> they do in Emacs. I would prefer to have a local binding, with cl-flet, but 
> this
> does not work in Emacs:
>
> (defun my-read-string (prompt)
>   (let ((delta 20 )
> (minibuffer-mode-map org-mode-map))
> (window-resize (minibuffer-window) delta)
> (cl-flet ((org-ctrl-c-ctrl-c ()
> (interactive)
> (let ((s (buffer-string)))
>   (exit-minibuffer) s))
>   (minibuffer-mode () #'org-mode)
>   (minibuffer-complete-and-exit () #'org-return)
>   (org-kill-note-or-show-branches () #'keyboard-escape-quit))
>   (read-string (concat "# Press C-c C-c to continue, C-c C-k to cancel\n# 
> "
>   prompt "\n\n")

Yes. cl-flet looks safe to me :-)

>
> Hooks serve a different purpose. Advice can serve same purpose with exactly
> same side effect, and some other limitations. With some care, let-binding is
> still more "local" then advice. With other words, I agree with you about the
> problems, but not with dogmatic approach that it should never be done, and
> that hooks and advices are the replacement.

Sorry if my words sounding dogmatic.
Else, I agree too :-)


>>
>>> I am very interested to hear more on the topic, since I would definitely 
>>> like to
>>> learn more about different techniques.
>>
>> Variables are designed to be overriden (let bounds). Functions are not
>
> I have never heard before that functions are not designed to be overriden. I
> think of them as two slots in a symbol structure; let creates bindings for 
> value
> slot, and flet for function slot. Functions are just objects or data as any
> other value in lisp.
>
>> (as there is only one binding at any given time).
>
> Yes, unfortunately, in Emacs it is so;

ok. We do really agree then :-)


> but I don't think it should be > :).

... oh no ! ;-)


>
> There is an interesting package by Nick Ferrier
>
> https://github.com/nicferrier/emacs-noflet

> but it does not seem to work, at least not for me.

It's almost like a temporary advice ;-)


About your use case, if what you need is asynchronous editing, maybe the
with-editor package will be of interest to you:
https://github.com/magit/with-editor/blob/main/lisp/with-editor.el

It allows sub-processes to call Emacs for editing tasks. It's used by
magit. It's easy enough to reuse. I've attached my attempt at it if
you're interested.

best,

Bruno

(cl-defun my-edit-async (finish &key mode buffer-name setup cancel)
  "Open a buffer, let the user edit its content.
Return the editing buffer.  Call FINISH in the editing buffer if
the user validates his edit (C-c C-c).  When CANCEL is non-nil,
call CANCEL in the editing buffer if the user cancels his
edit (C-c C-k). When done, delete the buffer and its content.

When MODE is non-nil, use it as the major-mode.  When BUFFER-NAME
is non-nil, use it to generate a new unique name of the editing buffer.
When SETUP is non-nil, call it in the edit buffer to setup the

Re: Problem with let/cl-letf binding stuff with org-capture

[Bruno Barbier]

Arthur Miller  writes:

> Bruno Barbier  writes:
>
>> Arthur Miller  writes:
>>
>> The hook `org-capture-mode-hook' will be run in your special
>> capture buffer. You can override the "C-c C-c" binding only there.
>
> Yes and in every other capture buffer

No. If you modify the hook only during your call to 'org-capture', it
will be called only once in your buffer.

>> Even if I could let bind the function at the right time, I would avoid
>> that solution, as I can't garantuee that this global hack will not break
>> other parts of Emacs (other captures, output filters, threads, timers,
>> etc.).
>
> Why do you think it will break other parts? This is not a global hack, on 
> contrary it
> exactly tries to prevent to be "global" in entire Emacs, by let-binding a name
> to a local lambda, which becomes "global" only in that buffer. If that
> explains.

You are assigning a local value to the global binding. Everything in
Emacs will use your functions until you exit the cl-letf. It's like if
you were using 'fset'.

Here is an example that uses cl-letf. Note that the call to
async-shell-command is outside the "local" binding, yet, the cl-letf
breaks it. You should try this in an other Emacs, just in case.

(defun oops ()
  (let ((v (async-shell-command "date" "!sh async")))
(cl-letf
(((symbol-function 'comint-output-filter)
  (lambda (proc string)
(message "async-shell-command is using my binding: %s" string)
(read-string "What's the password?"
  (read-string "what: ")
  )))
(oops)


>
> Here is another version on the same theme, where I don't think you could 
> modify the local
> environment without let-binding at all:
>
> #+begin_src emacs-lisp
> (defun my-read-string (prompt)
>   (let ((delta 20 )
> (minibuffer-mode-map org-mode-map))
> (window-resize (minibuffer-window) delta)
> (cl-letf (((symbol-function 'org-ctrl-c-ctrl-c)
>(lambda ()
>  (interactive)
>  (let ((s (buffer-string)))
>(exit-minibuffer) s)))
>   ((symbol-function 'minibuffer-mode) #'org-mode)
>   ((symbol-function 'minibuffer-complete-and-exit) #'org-return)
>   ((symbol-function 'org-kill-note-or-show-branches) 
> #'keyboard-escape-quit))
>   (read-string (concat "# Press C-c C-c to continue, C-c C-k to cancel\n# 
> " prompt "\n\n")
> #+end_src

I hope I've convinced you to not do that. I definitely will not try it,
as Emacs needs a working minibuffer for plenty of things: debugging,
saving, quitting, etc.


> read-string is written in C and creates its own minibuffer, which is deleted 
> by
> the time read-string exits. I don't know of any other way to cutomize exactly
> *that* minibuffer, without installing a hook or advising some functions, 
> which I
> think is way less clean and much more "global" than just running the function 
> in
> a local environment. As I understand, let binding for this purpose is a normal
> technique in lisps, but I am not an expert as said; I am actually 
> experimenting
> with this for the purpose of learning and seeing what is possible.

Yes, let binding is fundamental. But I think it's the first time I see
'cl-letf' with the 'symbol-function' place.


> ...
> but I am not sure if I can do anything here without introducing at-least an
> extra keymap, to not install into the org-capture-mode-map, so I can as well
> create a minor mode, but at this point it is not much different than
> re-invinting the read-string, so I'll terminate my experiment here :).

You can replace the buffer keymap with a keymap that only contain your custom
keys, and inherits everything else from org-capture-mode-map.


>
> But I wouldn't speak in some generic terms like "use hooks" or "advise" 
> instead
> of let-binding.

I didn't mean to say to not use let bindings. I'm trying to say that
using 'fset' (i.e. cl-letf + the symbol-function place) looks like a
really bad idea to me. And, in this case, hooks and adivces are what is
usually used.



> (defun org-project-new-project ()
>   (interactive)
>   (let ((org-capture-templates org-project-templates))
> (org-capture)))
>
> ...

> I don't know what would be the alternative, but let-binding on
> org-capture-templates, let me clearly re-use the functionality without 
> polluting
> the global org-capture-templates variable and without re-implementing pretty
> much anything.

T

Re: export to html, colored dates: org-mime export works but org to html does not

[Bruno Barbier]

Uwe Brauer  writes:

>>>> "BB" == Bruno Barbier  writes:
>
>> Uwe Brauer  writes:
>> ..
>
>>> (add-to-list 'org-export-filter-plain-text-functions 'my-html-red)
>> ..
>
>>> But if I have 
>>> #<2023-02-11>#
>>> in an org file and export it to html, no color is added to the timestamp.
>>> 
>>> I don't know how to debug this.
>>> 
>>> Any idea?
>
>> In org, "<2023-02-11>" is a date, not plain text. Thus, your function
>> doesn't get called with a text matching your date.
>
>> Note that using macros, you could get almost what you want. 
>
>> #+MACRO: IDATE @@html:$1@@
>
>> {{{IDATE(<2023-02-11>)}}}
>
>> And maybe ther is a way to also customize date format to get rid of the
>> brackets.
>
> Thanks, did you test it?
> I am asking because for me


I even did! :-)

You need to call the macro named 'IDATE' like this:

 * Title {{{IDATE(2023-02-11)}}}
 {{{IDATE(2023-02-11)}}}

See:
   (info "(org) Macro Replacement")

And it will work for you too.

Bruno
   

>
> #+LANGUAGE: 
> #+EXPORT_FILE_NAME: /home/oub/Desktop/testdate.html
> #+OPTIONS: toc:nil  
> #+HTML_HEAD_EXTRA:  body {font-size:150%}
> #+MACRO: IDATE @@html:$1@@
>
> * Title {{{<2023-02-11>}}}
> {{{(<2023-02-11>)}}}
>
>
> Does not work when exporting to html, and I have recent developer org
> version installed
>
> Uwe 
> -- 
> Warning: Content may be disturbing to some audiences
> I strongly condemn Putin's war of aggression against the Ukraine.
> I support to deliver weapons to Ukraine's military. 
> I support the ban of Russia from SWIFT.
> I support the EU membership of the Ukraine. 
> https://addons.thunderbird.net/en-US/thunderbird/addon/gmail-conversation-view/