So I have a patch written that doesn't completely fix the problem, but
maybe makes enough progress that someone else can figure it out.
The issue is that comint-prompt-regexp is reading the "% " as a prompt,
and taking everyone off before it. I've added another parameter to
org-babel-comint-with-output in the "meta" to pass our own regexp to
replace comint-prompt-regexp, which works except when it doesn't. The
regexp I've added is just "\n" now, so the newline characters are removed.
The tests that Daniele added as a patch don't quite pass though with
this. The issue is that something in the way the output is posted in the
output buffer includes the prompt occasionally is included in a line and
occasionally not. It seems the first time the block of code is executed,
it is included (and therefore needs to be removed) and each other time
it is not, so it is only the first time that it is run is not working
properly.
This is obviously way too fragile to actually merge, but I was hoping
the work I've done so far is enough to help someone else make progress.
I'll probably take another stab at this tomorrow (since it's bugging
me), but thought I'd share what I have for now.
Cheers,
Nick
On 5/6/21 8:24 AM, Ihor Radchenko wrote:
"Nicholas Savage" <n...@nicksavage.ca> writes:
I can confirm this too on the latest master.
I took a quick peek this morning, and my suspicion is that the problem is
somewhere within org-babel-comint-with-output in lisp/ob-comint.el, but I'm not
positive at this point.
I confirm as well. I also saw an anomaly in the comint buffer. Note that
all the output lines, except "five 0% six" are after the shell prompt.
As I remember, the code expects the result to be exactly at the prompt
line. So, for some reason the first command ("echo five 0% six") of the
second block does not get inserted at the empty line.
echo one 0% two
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ one 0%
two
echo tree 0% four
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ tree
0% four
echo 'org_babel_sh_eoe'
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $
org_babel_sh_eoe
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ echo
five 0% six
five 0% six
echo seven 0% eight
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ seven
0% eight
echo 'org_babel_sh_eoe'
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $
org_babel_sh_eoe
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $
Best,
Ihor
>From 6c7d39bfb9be38b54d23fcffbb09f1fcb96751f4 Mon Sep 17 00:00:00 2001
From: Nicholas Savage <n...@nicksavage.ca>
Date: Thu, 6 May 2021 19:17:33 -0400
Subject: [PATCH] ob-shell.el: Fix bug where shell output was incorrectly
truncated on special characters.
* lisp/ob-comint.el (org-babel-comint-with-output): Add fifth meta
optional argument for providing a custom prompt regexp.
lisp/ob-shell.el (org-babel-sh-evaluate): Implements using new
argument to prevent incorrect truncation on special characters.
If shell output included special characters that also occasionally are
included in shell prompts, such as "#" or "%", a regexp was tripping
up on them and cutting out part of the line. As ob-shell already cuts
out the shell prompts, this was not necessary and instead we can just
use \n as the separator. Original functionality was retained for other
ob-* files in case this was necessary.
---
lisp/ob-comint.el | 20 ++++++++++++--------
lisp/ob-shell.el | 2 +-
2 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 18d4f3c93..27ad6efd7 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -57,13 +57,14 @@ executed inside the protection of `save-excursion' and
(defmacro org-babel-comint-with-output (meta &rest body)
"Evaluate BODY in BUFFER and return process output.
-Will wait until EOE-INDICATOR appears in the output, then return
-all process output. If REMOVE-ECHO and FULL-BODY are present and
-non-nil, then strip echo'd body from the returned output. META
-should be a list containing the following where the last two
-elements are optional.
+Will wait until EOE-INDICATOR appears in the output, then return all
+process output. If REMOVE-ECHO and FULL-BODY are present and non-nil,
+then strip echo'd body from the returned output. PROMPT-REGEXP is a
+filter that, if provided, overrides the default regexp that tries to
+filter out the shell prompt. META should be a list containing the
+following where the last three elements are optional.
- (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY)
+ (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY PROMPT-REGEXP)
This macro ensures that the filter is removed in case of an error
or user `keyboard-quit' during execution of body."
@@ -71,7 +72,10 @@ or user `keyboard-quit' during execution of body."
(let ((buffer (nth 0 meta))
(eoe-indicator (nth 1 meta))
(remove-echo (nth 2 meta))
- (full-body (nth 3 meta)))
+ (full-body (nth 3 meta))
+ (prompt-regexp (if (nth 4 meta)
+ (nth 4 meta)
+ comint-prompt-regexp)))
`(org-babel-comint-in-buffer ,buffer
(let* ((string-buffer "")
(comint-output-filter-functions
@@ -111,7 +115,7 @@ or user `keyboard-quit' during execution of body."
"\n" "[\r\n]+" (regexp-quote (or ,full-body "")))
string-buffer))
(setq string-buffer (substring string-buffer (match-end 0))))
- (split-string string-buffer comint-prompt-regexp)))))
+ (split-string string-buffer ,prompt-regexp)))))
(def-edebug-spec org-babel-comint-with-output (sexp body))
(defun org-babel-comint-input-command (buffer cmd)
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 3eed0c164..9ec0425cb 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -263,7 +263,7 @@ return the value of the last statement in BODY."
#'org-trim
(butlast
(org-babel-comint-with-output
- (session org-babel-sh-eoe-output t body)
+ (session org-babel-sh-eoe-output t body "\n")
(dolist (line (append (split-string (org-trim body) "\n")
(list org-babel-sh-eoe-indicator)))
(insert line)
--
2.20.1
