Re: Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt)

[Matt]

  On Fri, 24 Mar 2023 07:38:58 -0400  Ihor Radchenko  wrote --- 

  > I suggest the following:
 > 1. Introduce a new customization `org-babel-default-shell', defaulting
 >to (or (executable-find "sh") (executable-find "cmd.exe")).
 > 2. Use the value as default shell in "shell" code blocks.
 > 3. Document and announce the change.
 > 4. Create org-lint checker that will mark "shell" code blocks as not
 >desired.
 > 
 > The above steps will ensure minimal breakage for existing uses of
 > "shell" blocks. Only users who wrote shell blocks for non-standard shell
 > will have to adapt.
 
These are good suggestions.  Thank you!

Re: Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt)

[Ihor Radchenko]

Samuel Wales  writes:

> i have a vague memory of having used sh and then we were told to use
> shell.  can we all use begin src sh now?

You can use anything listed in `org-babel-shell-names'.
Also, see https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-shell.html

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt)

[Samuel Wales]

i have a vague memory of having used sh and then we were told to use
shell.  can we all use begin src sh now?

On 3/24/23, Ihor Radchenko  wrote:
> Matt  writes:
>
>> What benefit does "shell" provide?
>>
>> - The "shell" language allows an arbitrary executable to be run.  This
>> means that shells other than those given in `org-babel-shell-names' can be
>> run.  People using a non-supported shell could still benefit from
>> ob-shell.
>>
>> What downsides does "shell" bring?
>>
>> - "shell" falls back to `shell-file-name' which can be an arbitrary
>> executable.  Whenever I hear "runs an arbitrary executable", my ears perk
>> up and I start to sweat.  There may be security considerations.
>> - If that executable is a shell, then the prompt gets set independently
>> from Emacs.  For the prompt to be filtered from the output, users would
>> need to provide Emacs with the correct regexp.  A recent thread discussed
>> creating a header arg to address this:
>> https://list.orgmode.org/87ttzgeg3w.fsf@localhost/
>> - We would get bug reports about non-supported shells which kind of work,
>> but have issues because they're not supported
>> - Maintence associated with supporting arbitrary (shell) executables
>>
>> As the current maintainer of ob-shell, I'm in favor of removing "shell" as
>> a Babel language.  The cons appear to far outweigh the pros.  However, I'm
>> aware others may have good use for it.  It's been a part of Org for nearly
>> a decade.  I'm sure it's part of people's workflow, especially since it's
>> been in the manual for 6 years.  Are there any pros, cons, use-cases, or
>> considerations I've overlooked?
>
> I would not see arbitrary executable to be such a big deal. At the end,
> if SHELL is set to something fishy, the user is likely in serious
> trouble anyway. SHELL is a part POSIX standard at the end.
>
> Yet, the problem with unsupported shells is indeed real.
> Moreover, "shell" code blocks are currently not portable to different
> environments.
>
> I suggest the following:
> 1. Introduce a new customization `org-babel-default-shell', defaulting
>to (or (executable-find "sh") (executable-find "cmd.exe")).
> 2. Use the value as default shell in "shell" code blocks.
> 3. Document and announce the change.
> 4. Create org-lint checker that will mark "shell" code blocks as not
>desired.
>
> The above steps will ensure minimal breakage for existing uses of
> "shell" blocks. Only users who wrote shell blocks for non-standard shell
> will have to adapt.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at .
> Support Org development at ,
> or support my work at 
>
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt)

[Ihor Radchenko]

Matt  writes:

> What benefit does "shell" provide?
>
> - The "shell" language allows an arbitrary executable to be run.  This means 
> that shells other than those given in `org-babel-shell-names' can be run.  
> People using a non-supported shell could still benefit from ob-shell.
>
> What downsides does "shell" bring?
>
> - "shell" falls back to `shell-file-name' which can be an arbitrary 
> executable.  Whenever I hear "runs an arbitrary executable", my ears perk up 
> and I start to sweat.  There may be security considerations.  
> - If that executable is a shell, then the prompt gets set independently from 
> Emacs.  For the prompt to be filtered from the output, users would need to 
> provide Emacs with the correct regexp.  A recent thread discussed creating a 
> header arg to address this: https://list.orgmode.org/87ttzgeg3w.fsf@localhost/
> - We would get bug reports about non-supported shells which kind of work, but 
> have issues because they're not supported
> - Maintence associated with supporting arbitrary (shell) executables
>
> As the current maintainer of ob-shell, I'm in favor of removing "shell" as a 
> Babel language.  The cons appear to far outweigh the pros.  However, I'm 
> aware others may have good use for it.  It's been a part of Org for nearly a 
> decade.  I'm sure it's part of people's workflow, especially since it's been 
> in the manual for 6 years.  Are there any pros, cons, use-cases, or 
> considerations I've overlooked?

I would not see arbitrary executable to be such a big deal. At the end,
if SHELL is set to something fishy, the user is likely in serious
trouble anyway. SHELL is a part POSIX standard at the end.

Yet, the problem with unsupported shells is indeed real.
Moreover, "shell" code blocks are currently not portable to different
environments.

I suggest the following:
1. Introduce a new customization `org-babel-default-shell', defaulting
   to (or (executable-find "sh") (executable-find "cmd.exe")).
2. Use the value as default shell in "shell" code blocks.
3. Document and announce the change.
4. Create org-lint checker that will mark "shell" code blocks as not
   desired.

The above steps will ensure minimal breakage for existing uses of
"shell" blocks. Only users who wrote shell blocks for non-standard shell
will have to adapt.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [SUGGESTION] ob-shell async result output should not contains shell prompt

[Ihor Radchenko]

Matt  writes:

> Changing the `ob-shell-async-chunk-callback' like this will fix it:
>
> @@ -276,7 +276,7 @@ See `org-babel-comint-async-indicator'.")
>  (defun ob-shell-async-chunk-callback (string)
>"Filter applied to results before insertion.
>  See `org-babel-comint-async-chunk-callback'."
> -  (replace-regexp-in-string comint-prompt-regexp "" string))
> +  (replace-regexp-in-string (concat (regexp-quote org-babel-sh-prompt) " *") 
> "" string))

This is trying to replicate what `org-babel-comint-with-output' does
already and is stumbling upon the same edge cases.

May you instead factor out the filtering code from
`org-babel-comint-with-output' and reuse it in ob-shell?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt)

[Matt]

 > Matt m...@excalamus.com> writes:
 >
 > > Is there a reason you're using "shell" instead of one of the shells listed 
 > > in `org-babel-shell-names'?

I'm still curious why you're using "shell".  I want to know if it's something 
you're using for a specific reason.  There's no wrong answer!

I ask because I have an agenda: as far as I can tell, "shell" as a Babel 
language is a historical accident.  

#+begin_longwinded_explanation
Originally, ob-shell.el was called ob-sh.el.  The main function was called 
`org-babel-execute:sh' and only /usr/bin/env sh was supported.  Over time it 
became clear that to support other shells, the "sh" name shouldn't be used for 
the package or the main function.  That is, 'sh' refers to a specific binary 
and, if other binaries such as bash, dash, csh, etc. were to be supported, it 
would be misleading for the Babel language to refer to a specific shell, 'sh'.  
So, the terminology was changed to something more general, "shell".  The 
package was renamed to "ob-shell.el", the "namespace" updated to "shell" (for 
example, `org-babel-execute:shell'), and the package load call changed from (sh 
. t) to (shell . t).  This officially happened with Org 8.2 (ORG-NEWS noted the 
change in commit 1a9adcb6d34bcbdff45445c827ed99dbf0b8, Tue Jan 21 09:52:31 
2014).  I think this gave people the (understandable) impression that "shell" 
was a valid Babel language, in addition to those listed in 
`org-babel-shell-names'.  

And this is where the accident happened: "shell" as a Babel language only 
**happens**to work.  The Babel framework looks for a function prototype like 
"org-babel-execute:".  When ob-sh.el was changed to ob-shell.el, the 
function `org-babel-execute:sh' became `org-babel-execute:shell'.   A call like 
follows is perfectly legal as far as the Babel framework is concerned:

#+begin_src shell
echo "hello, world"
#+end_src

When such a block is run, Babel looks for a function called 
`org-babel-execute:shell'.  Running the block prior to Org 8.2 should have 
failed because no `org-babel-execute:shell' function existed.  The name change 
happened to source Fri Dec 13 09:52:05 2013 in commit 
7a6c0e35415c4a173d101336029262f3a09abb91.  After the name change, the function 
existed and a block using "shell" would execute!  

The "shell" language specifier, as far as I can tell, was never really 
intentionally supported.  Instead, it just happened to work.  It happened to 
work because, as far back as the first org-babel-sh.el commit, the process 
buffer is created using the `shell' function.  I don't know the history of 
`shell', but presently the documentation says,

Program used comes from variable ‘explicit-shell-file-name’,
 or (if that is nil) from the ESHELL environment variable,
 or (if that is nil) from ‘shell-file-name’.

That is, the `shell' command falls back to `shell-file-name'.  I assume that 
`shell' has always had that, or a similar, fallback.  The `shell-file-name' is 
a direct path to an executable.   This means that when "shell" is used for the 
language, `shell-file-name' is called and **any** startup script, such as 
.bash_profile or .bashrc, is called.  The prompt could be set to **anything** 
and Emacs will never know, and can never know, what the prompt is without the 
user explicitly informing Emacs.

Aside from the code change which allowed "shell" to work, "official" support of 
"shell" comes from Org manual commit 9d072ad67517875069f913315d762c9bb1e9c3ec, 
Sun Dec 17 11:06:05 2017 (for example, 
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/doc/org-manual.org?id=f7aa8c19f5170dbf09538686fb569f9b60acbd6c#n18410).
  This appears unconnected with the code change.  The addition to the manual 
happened 4 years after the code name change and none of the commit messages 
around the time of code change suggest that "shell" was intended to work as a 
language.  In fact, I found this email from Eric Schulte (creator of Babel and 
maintainer at the time of the code change) which suggests that "shell" is in 
fact not supported or intented as a language 
(https://lists.gnu.org/archive/html/emacs-orgmode/2013-12/msg00294.html):

In response to the statement,

"a coworker used "#+BEGIN_SRC shell" where he should have written "#+BEGIN_SRC 
sh"

Eric says,

"[The suggested work around] would protect against this particular error"

#+end_longwinded_explanation

Regardless of whether "shell" was intended to work as a Babel language, the 
fact remains that it does work and that it's been advertised in the manual (at 
least) for 6 years.  What are the pros and cons of "shell"?

What benefit does "shell" provide?

- The "shell" language allows an arbitrary executable to be run.  This means 
that shells other than those given in `org-babel-shell-names' can be run.  
People using a non-supported shell could still benefit from ob-shell.

What downsides does "shell" bring?

- "shell" falls back to `shell-file-name' which can be an arbitrary executable. 
 

Re: [SUGGESTION] ob-shell async result output should not contains shell prompt

[Matt]

  On Thu, 23 Mar 2023 07:12:29 -0400  Christopher M. Miles  wrote --- 
 
 > #+begin_src bash :session "*ob-shell-bash*" :async t
 > sleep 30
 > echo "hello, world"
 > #+end_src
 > 
 > #+RESULTS[(2023-03-23 19:14:15) 23f9ad130f7a1268e21821c6baaea2b057c70d3e]:
 > : org_babel_sh_prompt> hello, world
 > 
 > Still got a prompt. Is this intended? I think the output should be kept 
 > clean because the result
 > output might be used as input for other source blocks as data.

I absolutely agree.  This is a bug.  

It should be an easy fix on my end but, in case there are details I'm 
overlooking, this specific example can use the following workaround:

#+begin_src bash :session "*ob-shell-bash*" :async t
sleep 30 && echo "hello, world"
#+end_src

To explain what's going on...

#+begin_longwinded_explanation
Shell output is filtered for prompts which should be removed before inserting 
the results.  The issue is that the filter assumes the prompt starts at the 
beginning of the line.  When sleep is called, it returns nothing and the next 
prompt appears on the same line:

sh-5.1$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2=
org_babel_sh_prompt> echo 
'ob_comint_async_shell_start_770d9c8f-deda-4359-aee9-a433a75a5e0d'
echo "1"
sleep 3
echo "2"
echo 'ob_comint_async_shell_end_770d9c8f-deda-4359-aee9-a433a75a5e0d'
ob_comint_async_shell_start_770d9c8f-deda-4359-aee9-a433a75a5e0d
org_babel_sh_prompt> 1
org_babel_sh_prompt> org_babel_sh_prompt> 2
org_babel_sh_prompt> 
ob_comint_async_shell_end_770d9c8f-deda-4359-aee9-a433a75a5e0d

Changing the `ob-shell-async-chunk-callback' like this will fix it:

@@ -276,7 +276,7 @@ See `org-babel-comint-async-indicator'.")
 (defun ob-shell-async-chunk-callback (string)
   "Filter applied to results before insertion.
 See `org-babel-comint-async-chunk-callback'."
-  (replace-regexp-in-string comint-prompt-regexp "" string))
+  (replace-regexp-in-string (concat (regexp-quote org-babel-sh-prompt) " *") 
"" string))
 
 (defun org-babel-sh-evaluate (session body  params stdin cmdline)
   "Pass BODY to the Shell process in BUFFER.
#+end_longwinded_explanation

Re: [SUGGESTION] ob-shell async result output should not contains shell prompt

[Christopher M. Miles]

Matt  writes:

>   On Wed, 22 Mar 2023 23:25:50 -0400  Christopher M. Miles  wrote --- 
>  > 
>  > The ob-shell async result output contains the shell prompt. I think it
>  > should not be captured.
>  > 
>  > #+begin_src shell :session "test2" :async t
>  > sleep 30
>  > echo "hello, world"
>  > #+end_src
>  > 
>  > #+RESULTS[(2023-03-23 11:19:22) 461ed5de684f6e619890709175ec73e80b67b2d6]:
>  > : bash-5.2$ hello, world
>
> Thanks for reporting this.
>
> Try using for the babel language whatever shell the variable 
> `shell-file-name' gives.  For example, if `shell-file-name' is /bin/bash, do 
> this:
>
> #+begin_src bash :session "test2" :async t
> sleep 1
> echo "hello, world"
> #+end_src
>
> Is there a reason you're using "shell" instead of one of the shells listed in 
> `org-babel-shell-names'?

Using language identities like bellowing:

#+begin_src sh :session "*ob-shell*" :async t
sleep 30
echo "hello, world"
#+end_src

#+RESULTS[(2023-03-23 19:14:12) dd777a237986481833c08eb5eceac717576eddb7]:
: org_babel_sh_prompt> hello, world

#+begin_src bash :session "*ob-shell-bash*" :async t
sleep 30
echo "hello, world"
#+end_src

#+RESULTS[(2023-03-23 19:14:15) 23f9ad130f7a1268e21821c6baaea2b057c70d3e]:
: org_babel_sh_prompt> hello, world

#+begin_src zsh :session "*ob-shell-zsh*" :async t
sleep 30
echo "hello, world"
#+end_src

#+RESULTS[(2023-03-23 19:14:17) 2bb44d96c2e482a90c5a89bdde0b64d0319663a1]:
: % 

 
:  
: % 

 
:  
: hello, world
: % 

 
:  

Still got a prompt. Is this intended? I think the output should be kept clean 
because the result
output might be used as input for other source blocks as data.

Maybe error in my Org babel settings? Bellowing is my system and variable 
values:

#+begin_src emacs-lisp
system-type
#+end_src

#+RESULTS[(2023-03-23 19:27:13) 7df8395169a77d83cb6a5a6efc2223d412813efa]:
: darwin

#+begin_src emacs-lisp
shell-file-name
#+end_src

#+RESULTS[(2023-03-23 19:26:33) e6fed18a9a543dd6320385ee715d9ee68b464a04]:
: /opt/homebrew/bin/bash

#+begin_src emacs-lisp
org-babel-sh-prompt
#+end_src

#+RESULTS[(2023-03-23 19:30:12) f6efc29dba5be2171eba0a25abec19908fb1c6be]:
: org_babel_sh_prompt> 

#+begin_src emacs-lisp
org-babel-shell-names
#+end_src

#+RESULTS[(2023-03-23 19:27:27) 360d6d35db3eb48deb664349eed34b7541923ca2]:
| sh | bash | zsh | fish | csh | ash | dash | ksh | mksh | posh |

#+begin_src emacs-lisp :results pp
org-babel-shell-set-prompt-commands
#+end_src

#+RESULTS[(2023-03-23 19:27:44) 910fbbafc6fea4a1846f5a31f8b7dd102eca4928]:
: (("fish" . "function fish_prompt\necho \"%s\"\nend")
:  ("csh" . "set prompt=\"%s\"\nset prompt2=\"\"")
:  ("posh" . "function prompt { \"%s\" }")
:  (t . "PROMPT_COMMAND=;PS1=\"%s\";PS2="))

-- 

[ stardiviner ]
I try to make every word tell the meaning that I want to express without 
misunderstanding.

Blog: https://stardiviner.github.io/
IRC(libera.chat, freenode): stardiviner, Matrix: stardiviner
GPG: F09F650D7D674819892591401B5DF1C95AE89AC3


signature.asc
Description: PGP signature

Re: [SUGGESTION] ob-shell async result output should not contains shell prompt

[Matt]

  On Wed, 22 Mar 2023 23:25:50 -0400  Christopher M. Miles  wrote --- 
 > 
 > The ob-shell async result output contains the shell prompt. I think it
 > should not be captured.
 > 
 > #+begin_src shell :session "test2" :async t
 > sleep 30
 > echo "hello, world"
 > #+end_src
 > 
 > #+RESULTS[(2023-03-23 11:19:22) 461ed5de684f6e619890709175ec73e80b67b2d6]:
 > : bash-5.2$ hello, world

Thanks for reporting this.

Try using for the babel language whatever shell the variable `shell-file-name' 
gives.  For example, if `shell-file-name' is /bin/bash, do this:

#+begin_src bash :session "test2" :async t
sleep 1
echo "hello, world"
#+end_src

Is there a reason you're using "shell" instead of one of the shells listed in 
`org-babel-shell-names'?

[SUGGESTION] ob-shell async result output should not contains shell prompt

[Christopher M. Miles]

The ob-shell async result output contains the shell prompt. I think it
should not be captured.

#+begin_src shell :session "test2" :async t
sleep 30
echo "hello, world"
#+end_src

#+RESULTS[(2023-03-23 11:19:22) 461ed5de684f6e619890709175ec73e80b67b2d6]:
: bash-5.2$ hello, world

-- 

[ stardiviner ]
I try to make every word tell the meaning that I want to express without 
misunderstanding.

Blog: https://stardiviner.github.io/
IRC(libera.chat, freenode): stardiviner, Matrix: stardiviner
GPG: F09F650D7D674819892591401B5DF1C95AE89AC3


signature.asc
Description: PGP signature