Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)

[Nathaniel Nicandro]

Ihor Radchenko  writes:

> Ihor Radchenko  writes:
>
>> Nathaniel Nicandro  writes:
>>
>>> Feedback appreciated!
>>
>> Thanks for the update!
>> ...
>>> I've finally implemented a solution to what I've discussed previously,
>> ...
>
> It has been a while since the last update in this thread.
> Nathaniel, may I know if you are still working on this?

Hello Ihor,

Yes I'm still working on this.  Attached is an updated patch with some
tests this time.  It's still a work in progress.  Below are responses to
your previous comments about my last update and some comments about this
current patch.

> This is very fragile.
> I believe that hooking into `org-fold-check-before-invisible-edit'
> would lead to simpler implementation.

Thank you for the feedback.  I indeed was able to come up with a
more simpler solution by hooking into that function.

To integrate with `org-fold-check-before-invisible-edit' I had to
introduce two variables, `org-fold-visibility-detail' which is set to
the argument of `org-fold-show-set-visibility' when that function is
called and `org-ansi-fontify-begin' to determine the start of the
fontification region to see if it's close to the beginning of an
invisible sequence that should be turned visible.

Let me know if this is an OK approach.

I ran into an issue when trying to hook into
`org-fold-check-before-invisible-edit' in that when it revealed a
sequence at the end of a line, there would be an extra fontification
cycle that would occur after the reveal which would cause the sequence
to be re-hidden again.  To counteract this I had to use
`buffer-chars-modified-tick' in the way I do.  I couldn't figure out
why redisplay was causing that extra fontification cycle when there
were no modifications to the buffer.

> 1. Open the file and move to the end of the headline "Greater elements"
> 2.  
> 3. Observe fontification extending past the title.

This is fixed.  I think it was due to specifying the contents-end
position as the end of the region to highlight instead of the
line-end-position for headlines.

> I also edited it around in various places and I managed to trigger
> parser errors when the parser lost track of the modifications. This
> was presumably because your patch edited the buffer.

I no longer make edits to the buffer.  The ANSI sequences are no
longer accompanied by the zero width spaces from the idea that I had
before.

With this patch, editing around sequences should be more stable and
non-surprising.  Basically if a sequence is invisible around point and
you edit it, the sequence remains visible.  It is only after the first
edit outside of a sequence that should make the sequence invisible.
Whenever a sequence is being edited, it should always be visible and
not turn invisible while in the middle of editing it, e.g. due to an
invalid sequence turning valid.

Some comments about the patch, as it currently stands, follow.

- I've introduced two text properties `org-ansi' and
  `org-ansi-context'.

  The first is placed on the regions that actually contain ANSI
  sequences and holds information about the sequence that is useful to
  keep around to detect when a sequence has been modified or deleted
  between fontification cycles, as well as information about whether
  or not a sequence should be revealed due to modifications or because
  of visibility changes.

  The second property holds the ANSI context, as defined by
  `ansi-color-context-region', for regions that actually have been
  highlighted or processed by `org-ansi-process-region'.  Storing the
  ANSI context is done so that on fontifying some new region, the
  context that should be used can be determined simply by examining
  the property on an appropriate region before the start of the
  fontification.  The property is also used to determine the extent of
  a context or sequence, how far forward into the buffer its effects
  last.  The extent of a context is useful for extending the region
  being fontified to include the extent of a sequence which has been
  modified or deleted between fontification cycles.

  Currently I only extend the fontification region to include the
  extent when there has been a deletion or modification of a sequence
  in the region up for fontification (`org-ansi-extend-region').  I've
  not found a way to extend the fontification to a region including
  the full extent of a newly inserted sequence, in such cases the code
  as it stands now will fontify past the limit of fontification to the
  end of the element.

- The `org-ansi-process-*' functions boil down to calls to
  `org-ansi-process-region' which does the actual highlighting and
  bookkeeping of text properties on the regions.  Each of the process
  functions are just aware of the varying types of element structure
  in an Org document.  They are supposed to process an element's
  region from point to some limit or to the end of the element,
  applying properties to the highlightable regions.  If it's to the
  end of the 

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Max Nikulin  writes:

>> I guess not on Windows (bug#71081). But otherwise I agree.
>
> It was for starting mailcap viewer and Windows was not an issue.

bug#71081 is about shell-command-on-region, not about mailcap.
(and pty may actually not matter there, because Windows' cmd.exe is not
even POSIX-compient...)

> Some commands try to read standard input if it is available.  In 
> interactive sessions it may be hidden due to time interval between typed 
> commands.  Be careful when they are used in Org source blocks.
> [[https://mywiki.wooledge.org/BashFAQ/089][BASH FAQ #89]]
> warns concerning =ssh= and =ffmpeg=.  Either explicitly specify
> =
> #+begin_example
> ssh example.org 'sed -i -e s/foo/bar/ file.txt'  ssh example.org 'tee >>file.log' <<"EOF"
> Added by org-babel
> EOF
> #+end_example
>
> (Quotes around "EOF" suppress variable expansion in the text.)  Tools 
> may have dedicated options, for example =ssh -n= is a more concise way 
> to avoid the pitfall.

May you submit a patch for the WORG docs that documents this caveat?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 01/07/2024 16:51, Ihor Radchenko wrote:

Max Nikulin writes:


My experience is that it is not easy to start a process without a tty in
Emacs. Eli will object.


I guess not on Windows (bug#71081). But otherwise I agree.


It was for starting mailcap viewer and Windows was not an issue.


Although, I am not sure what we need to document and what we just need
to rewrite.


I have posted it to this thread and to https://debbugs.gnu.org/67259

Some commands try to read standard input if it is available.  In 
interactive sessions it may be hidden due to time interval between typed 
commands.  Be careful when they are used in Org source blocks.

[[https://mywiki.wooledge.org/BashFAQ/089][BASH FAQ #89]]
warns concerning =ssh= and =ffmpeg=.  Either explicitly specify
=>file.log' <<"EOF"
Added by org-babel
EOF
#+end_example

(Quotes around "EOF" suppress variable expansion in the text.)  Tools 
may have dedicated options, for example =ssh -n= is a more concise way 
to avoid the pitfall.



In theory, we may work around all the pty-related problems
by simply running a script file (that possibly also echoes its commands
to keep them recorded for user examination).


Ideally some kind of bracketed paste should be added to `shell', 
otherwise it is insecure. However it is not an option for dash. I still 
have slight hope that implementation of =C-x C-e= might give another 
idea. Some subtle issues may arise with "source", but perhaps they are 
more rare than naive ssh usage. If you know how to add text to shell 
buffer without sending it to the process then you may try. I would leave 
an option to switch back to "direct" input.

Re: [FR] :noweb-wrap header arg

[Ihor Radchenko]

Ihor Radchenko  writes:

>> Let me know what you think!
>
> Consider cases like
> ...

It has been over a month since the last message in this thread.
Amy, may I know if you are still working on the patch?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Max Nikulin  writes:

 I do not see a great benefit of re-implementing session support via
 actual terminal. (I think we agree here).
>>>
>>> `shell' and so comint allocates "actual" pty.
>> 
>> Are you sure?
> ...
>   123454 pts/0Sl 0:16  \_ emacs -Q
>   125104 pts/4Ss+0:00  \_ /bin/bash --noediting -i
>   129807 pts/5Ss+0:00  \_ /bin/bash
>   132172 pts/6Ss+0:00  \_ /usr/bin/bash --noediting -i
>
> pts/0 is created by konsole, others are `shell' and `term' sessions 
> inside Emacs.
>
> My experience is that it is not easy to start a process without a tty in 
> Emacs. Eli will object.

I guess not on Windows (bug#71081). But otherwise I agree.

Although, I am not sure what we need to document and what we just need
to rewrite. In theory, we may work around all the pty-related problems
by simply running a script file (that possibly also echoes its commands
to keep them recorded for user examination).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 30/06/2024 23:42, Ihor Radchenko wrote:

Max Nikulin writes:


On 30/06/2024 22:28, Ihor Radchenko wrote:

I do not see a great benefit of re-implementing session support via
actual terminal. (I think we agree here).


`shell' and so comint allocates "actual" pty.


Are you sure?


ps wf
PID TTY  STAT   TIME COMMAND
   2269 pts/0Ss 0:01 /bin/bash
 123454 pts/0Sl 0:16  \_ emacs -Q
 125104 pts/4Ss+0:00  \_ /bin/bash --noediting -i
 129807 pts/5Ss+0:00  \_ /bin/bash
 132172 pts/6Ss+0:00  \_ /usr/bin/bash --noediting -i

pts/0 is created by konsole, others are `shell' and `term' sessions 
inside Emacs.


My experience is that it is not easy to start a process without a tty in 
Emacs. Eli will object.