Re: We have asynchronous sessions, why have anything else?
Ivar Fredholm writes: > Hi Ihor, I have a prototype of what I mentioned earlier, at least for python. > This supports asynchronous, synchronous, session, and session-less blocks. > It's pretty messy but it helps to illustrate what I had in mind. Let me know > what you think. I am not sure how I feel about it. >From cursory look, the idea looks reasonable implementation-wise. However, there is one big important requirement which does not appear to be obeyed by your code: We _must_ be backwards-compatible. All the existing babel backends must not be broken, especially third-party ones. We must not make breaking changes to non-private function definitions. Also, I do not see error handling as it is implemented in our current babel code: Errors should be displayed in a popup buffer. Best, Ihor
Re: We have asynchronous sessions, why have anything else?
Hi Ihor, I have a prototype of what I mentioned earlier, at least for python. This supports asynchronous, synchronous, session, and session-less blocks. It's pretty messy but it helps to illustrate what I had in mind. Let me know what you think. Sent with Proton Mail secure email. --- Original Message --- On Monday, June 27th, 2022 at 4:56 AM, Ihor Radchenko wrote: > Ivar Fredholm freddyho...@protonmail.com writes: > > > I believe the two could be unified if we expand the functionality of > > the async filter to look for 'exception' tags. Then each language > > implementation must only put the org-babel src block in a try-except > > type construction and put the error message into the except block. > > > I am not even sure if all the babel backends support try-except. > Think about ob-gnuplot or, say, ob-latex. > > Best, > Ihor(defun eval-file (file) (with-temp-buffer (insert-file-contents file) (eval-buffer))) (eval-file "~/new_org/org-mode/lisp/ob-core.el") (eval-file "~/new_org/org-mode/lisp/ob-comint.el") (eval-file "~/new_org/org-mode/lisp/ob-python.el") (eval-file "~/new_org/org-mode/lisp/org-attach.el") (require 'subr-x) (require 'eieio) (require 'cl-lib) (defvar org-babel-session-list nil "List of all sessions") (defvar org-babel-shell-buffers nil "List of interpreter buffers. This gets garbage collected every time a source block is run. Any process-less buffer gets deleted.") (defclass latch () ((process :initform (start-process "latch" nil nil)) (value :initform nil)) :documentation "A blocking latch that can be used any number of times.") (cl-defmethod wait ((latch latch) timeout) "Blocking wait on LATCH for a corresponding `notify', returning the value passed by the notification. Wait at most TIMEOUT seconds (float allowed), returning nil if the timeout was reached with no input. The Emacs display will not update during this period but I/O and timers will continue to run." (accept-process-output (slot-value latch 'process) timeout) (slot-value latch 'value)) (cl-defmethod notify ((latch latch) value) "Release all execution contexts waiting on LATCH, passing them VALUE." (setf (slot-value latch 'value) value) (process-send-string (slot-value latch 'process) "\n")) (cl-defmethod destroy ((latch latch)) "Destroy a latch, since they can't be fully memory managed." (ignore-errors (delete-process (slot-value latch 'process (defun make-latch () "Make a latch which can be used any number of times. It must be `destroy'ed when no longer used, because the underlying process will not be garbage collected." (make-instance 'latch)) (defun destroy-all-latches () "Destroy all known latches." (cl-loop for process in (process-list) when (string-match-p "latch\\(<[0-9]+>\\)?" (process-name process)) do (delete-process process))) ;; Code for the administration of sessions and their processes. (defclass org-babel-session () ((name :initarg :name :documentation "Name of the session, should be unique on a per-language basis or 'none' if the associated source block is session-less.") (language :initarg :language :documentation "The language for the source block associated to this session.") (is-none :initform nil :documentation "Indicates whether we should delete the session once it has finished executing its source block.") (unique-id :initform "" :documentation "This is the unique process identifier for the session.") (process :initform nil :documentation "The interpreter or shell for the session.") (buffer :initform nil :documentation "The buffer associated with process") (ready-for-input :initform nil :documentation "A variable indicating whether the interpreter is ready to accept more input.") (input-latch :initform nil :documentation "A latch that blocks execution until the interpreter has finished processing the current input. This is used to emulate synchronous blocks using the asynchronous process filter.") (indicator-regexp :initform nil :documentation "Holds the indicator regexp that the async filter will look for in the comint output. The user must define this on a per-language basis by defining a `org-babel-async-indicator:LANG' constant.") (org-buffers :initform nil :documentation "A list of buffers to look through when searching for a place to insert the results of a source block.") (async :initform nil :documentation "Tell the process whether to notify its latch when ready for input or not") (current-dangling :initform "" :documentation "Holds the most recent text provided by the interpreter in case of output buffering.")) "To implement concrete classes of this class, one must first define: a session initializer which launches an interpreter, and a method to asynchronously send input to said interpreter. Language
Re: We have asynchronous sessions, why have anything else?
The Jupyter project is one approach to this. It currently has dozens of kernels for different languages, and new kernels can certainly be made. The emacs-jupyter package provides one implementation of an interface. It is complex, and relies on a compiled module for the zeromq message passing library. I am not advocating for this as the solution for org-babel, but it is an interesting case study. You can even connect to remote kernels. I use it a lot. On Mon, Jun 27, 2022 at 8:56 PM Tim Cross wrote: > > Tom Gillespie writes: > > >> I am not even sure if all the babel backends support try-except. > >> Think about ob-gnuplot or, say, ob-latex. > > > > Indeed many do not. Defining some standard "features" > > for org babel language implementations is something that > > is definitely of interest so that we can provide clear interfaces > > for things like stdio, error handling, return values, async, > > file output, remote execution, sessions, return value caching, > > module discovery/tangling, execution from file vs stdin, execution > > without a file system path, runtime environment specification, > > and much more. However, at the moment there is only a preliminary > > survey of a subset of these that was put together by Ian Martins. > > > > https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html > > > >> the two could be unified if we expand the functionality of the async > filter > > > > While this might be possible, I would definitely hold off on this because > > the changes in semantics absolutely will break many users' blocks. We > > barely knew what the impact of changing the default return value for > shell > > blocks would be. > > > > I absolutely look forward to the day when this can be done safely and > > with confidence, but I think we need a much stronger handle on babel > > interfaces in general before such a change could even be considered. > > > > At the moment each ob lang impl pretty much has to be considered > > to be completely unique, even if the text looks like bash for e.g. > > shell, comint, and screen. Users are known to rely on undocumented > > quirks of the ob lang impls that can differ wildly in their semantics. > > > > Well said Tom. > > As you point out, there are numerous deficiencies with the current > implementation, despite the fact it all sort of works. To get the sort > of improvements and consistency users want, I suspect this needs more > than just small tweaks around the edges. > > To some extent, I see the current babel implementation as similar to a > prototype. It has worked well and we have learnt a lot about what people > want to use it for and the type of functionality they are wanting and > what some of the core challenges are. Now comes the next step, which is > definitely non-trivial. We need to take all that knowledge and > consolidate it into a single model from which we can define the > interfaces and associated APIs. A big job which will take considerable > time. > > -- John --- Professor John Kitchin (he/him/his) Doherty Hall A207F Department of Chemical Engineering Carnegie Mellon University Pittsburgh, PA 15213 412-268-7803 @johnkitchin https://kitchingroup.cheme.cmu.edu https://pointbreezepubs.gumroad.com/ pycse bookstore
Re: We have asynchronous sessions, why have anything else?
Tom Gillespie writes: >> I am not even sure if all the babel backends support try-except. >> Think about ob-gnuplot or, say, ob-latex. > > Indeed many do not. Defining some standard "features" > for org babel language implementations is something that > is definitely of interest so that we can provide clear interfaces > for things like stdio, error handling, return values, async, > file output, remote execution, sessions, return value caching, > module discovery/tangling, execution from file vs stdin, execution > without a file system path, runtime environment specification, > and much more. However, at the moment there is only a preliminary > survey of a subset of these that was put together by Ian Martins. > > https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html > >> the two could be unified if we expand the functionality of the async filter > > While this might be possible, I would definitely hold off on this because > the changes in semantics absolutely will break many users' blocks. We > barely knew what the impact of changing the default return value for shell > blocks would be. > > I absolutely look forward to the day when this can be done safely and > with confidence, but I think we need a much stronger handle on babel > interfaces in general before such a change could even be considered. > > At the moment each ob lang impl pretty much has to be considered > to be completely unique, even if the text looks like bash for e.g. > shell, comint, and screen. Users are known to rely on undocumented > quirks of the ob lang impls that can differ wildly in their semantics. > Well said Tom. As you point out, there are numerous deficiencies with the current implementation, despite the fact it all sort of works. To get the sort of improvements and consistency users want, I suspect this needs more than just small tweaks around the edges. To some extent, I see the current babel implementation as similar to a prototype. It has worked well and we have learnt a lot about what people want to use it for and the type of functionality they are wanting and what some of the core challenges are. Now comes the next step, which is definitely non-trivial. We need to take all that knowledge and consolidate it into a single model from which we can define the interfaces and associated APIs. A big job which will take considerable time.
Re: We have asynchronous sessions, why have anything else?
> I am not even sure if all the babel backends support try-except. > Think about ob-gnuplot or, say, ob-latex. Indeed many do not. Defining some standard "features" for org babel language implementations is something that is definitely of interest so that we can provide clear interfaces for things like stdio, error handling, return values, async, file output, remote execution, sessions, return value caching, module discovery/tangling, execution from file vs stdin, execution without a file system path, runtime environment specification, and much more. However, at the moment there is only a preliminary survey of a subset of these that was put together by Ian Martins. https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html > the two could be unified if we expand the functionality of the async filter While this might be possible, I would definitely hold off on this because the changes in semantics absolutely will break many users' blocks. We barely knew what the impact of changing the default return value for shell blocks would be. I absolutely look forward to the day when this can be done safely and with confidence, but I think we need a much stronger handle on babel interfaces in general before such a change could even be considered. At the moment each ob lang impl pretty much has to be considered to be completely unique, even if the text looks like bash for e.g. shell, comint, and screen. Users are known to rely on undocumented quirks of the ob lang impls that can differ wildly in their semantics. Best! Tom
Re: We have asynchronous sessions, why have anything else?
Ivar Fredholm writes: > I believe the two could be unified if we expand the functionality of > the async filter to look for 'exception' tags. Then each language > implementation must only put the org-babel src block in a try-except > type construction and put the error message into the except block. I am not even sure if all the babel backends support try-except. Think about ob-gnuplot or, say, ob-latex. Best, Ihor
Re: We have asynchronous sessions, why have anything else?
Hi, Ihor, I believe the two could be unified if we expand the functionality of the async filter to look for 'exception' tags. Then each language implementation must only put the org-babel src block in a try-except type construction and put the error message into the except block. The async filter will then find the error information and report it back to the org-buffer. We could then emulate the synchronous session functionality by adding a filter to the async-filter which blocks until an 'end' or 'exception' tag is detected. This way, all four combinations of session/session-less and synchronous/asynchronous header arguments could be handled by the asynchronous code alone. Best, Guacho Sent with Proton Mail secure email. --- Original Message --- On Saturday, June 25th, 2022 at 10:28 PM, Ihor Radchenko wrote: > Ivar Fredholm freddyho...@protonmail.com writes: > > > A session-less block can be executed by starting a session with a special > > name (say "*none") which always gets killed after block execution is > > completed. For interpreter-less languages, we could use the shell as an > > interpreter (for instance, if we wanted to execute C, we could just start a > > shell, and send it the gcc command to compile and execute). Would this not > > cut down the amount of code that needs to be maintained and uniformize the > > existing code? > > > Feel free to compare ob-eval.el and ob-comint.el. Their functionality is > not equivalent. In particular ob-eval.el has a better handling of > errors. > > If you find a way to unify the two without loosing the functionality, it > will be welcome. > > Best, > Ihor
Re: We have asynchronous sessions, why have anything else?
Ivar Fredholm writes: > A session-less block can be executed by starting a session with a special > name (say "*none") which always gets killed after block execution is > completed. For interpreter-less languages, we could use the shell as an > interpreter (for instance, if we wanted to execute C, we could just start a > shell, and send it the gcc command to compile and execute). Would this not > cut down the amount of code that needs to be maintained and uniformize the > existing code? Feel free to compare ob-eval.el and ob-comint.el. Their functionality is not equivalent. In particular ob-eval.el has a better handling of errors. If you find a way to unify the two without loosing the functionality, it will be welcome. Best, Ihor
We have asynchronous sessions, why have anything else?
A session-less block can be executed by starting a session with a special name (say "*none") which always gets killed after block execution is completed. For interpreter-less languages, we could use the shell as an interpreter (for instance, if we wanted to execute C, we could just start a shell, and send it the gcc command to compile and execute). Would this not cut down the amount of code that needs to be maintained and uniformize the existing code? Sent with [Proton Mail](https://proton.me/) secure email.
