Re: [O] Proposal and RFC for improving ob-python
hi, John, > I like the idea behind ob-ipython, hopefully where it might go as > ob-jupyter, which could make it able to run many other languages via > the jupyter project and it's kernels. thanks for the reply. i'm not really up to speed on Jupyter, et al. but, from a brief look, my work is probably more directed at a more plain, vanilla python. were an [ij]python mode available, i can imagine being captivated it. but, if i were to try and hack on any code right now, it would probably be normal python mode. cheers, Greg
Re: [O] Proposal and RFC for improving ob-python
now I can not remember if I sent this earlier. I like the idea behind ob-ipython, hopefully where it might go as ob-jupyter, which could make it able to run many other languages via the jupyter project and it's kernels. The last time I tried it, it only supported sessions, and I had to add non-session support. I doubt it supported async running either. Before any major effort went in to a new ob-python, it is probably worth a serious look at what the jupyter project could offer. Greg Minshall writes: > hi. i recently got fed up with ob-python.el behavior w.r.t. :session > handling (startup issues; blank lines [really, a python problem, imho, > but a workaround maybe could work], and value scraping), looked, and saw > this discussion from last December, but nothing since then. has there > been any progress, either by Ondřej or anyone else? if my frustration > level remains high enough, i might try to see if i can do anything. > cheers, Greg Minshall -- Professor John Kitchin Doherty Hall A207F Department of Chemical Engineering Carnegie Mellon University Pittsburgh, PA 15213 412-268-7803 @johnkitchin http://kitchingroup.cheme.cmu.edu
Re: [O] Proposal and RFC for improving ob-python
hi. i recently got fed up with ob-python.el behavior w.r.t. :session handling (startup issues; blank lines [really, a python problem, imho, but a workaround maybe could work], and value scraping), looked, and saw this discussion from last December, but nothing since then. has there been any progress, either by Ondřej or anyone else? if my frustration level remains high enough, i might try to see if i can do anything. cheers, Greg Minshall
Re: [O] Proposal and RFC for improving ob-python
Ondřej Grover writes: [...] >> But what about when :results != value? Doesn't your proposal only >> handle returning the last value? >> > You mean :results output ? In that case it could just omit the > "open(...).write(...) " part and capture anything the console prints before > the primary prompt appears again. > Or it could capture stdout into a file, but I think that's needlessly > complicated: > > python -i << HEREDOC_END > import sys > sys.stdout = open() # replace stdout with some file > _ = block_eval(""" > > """) > sys.stdout.close() > sys.stdout = sys.__stdout__ # restore stdout > HEREDOC_END (I think you should rely on Org babel's shared utilities, e.g. org-babel-eval and org-babel-eval-read-file, unless there is a specific reason they won't do. At any rate, right now I'm just trying to get a picture of how block_eval would fit in.) So these are the main options to consider for Python: | | output | value | |-++---| | non-session | a | b | | session | c | d | a) Using block_eval here seems unhelpful here because the method in place is already simple: (org-babel-eval org-babel-python-command body) b) block_eval replaces the need for dumping the code into a function. Using block_eval in org-babel-python-wrapper-method would allow you to get rid of the odd "return" use. c) Currently, this is handled by inserting the body lines one at a time and then getting the output by splitting on the prompt. Aside from race condition issues, this has limitations with handling blank lines in the body. block_eval could help here because the whole body could be sent through as one line of input to the prompt. This should resolve most issues with prompt splitting, as well as lift the body formatting restrictions for sessions. However, the main issue I see with this is that I think it would lose the main distinction between session and non-session for ":results output" that is discussed in (info "(org) Results of evaluation"): #+BEGIN_SRC python :results output print "hello" 2 print "bye" #+END_SRC #+RESULTS: : hello : bye versus this #+BEGIN_SRC python :results output :session print "hello" 2 print "bye" #+END_SRC #+RESULTS: : hello : 2 : bye d) The variable _ is assigned the last value by the shell, so the current mechanism should work even if block_eval is used as described in c. I've attached a quick-and-dirty patch, just for discussion. At least under light testing, it resolves the issues I listed in my previous response [1]. It does have the change in behavior discussed in c, which I see as problematic. It also raises the issue of how to incorporate the block_eval code. I think there are two main options. * Require the user have the module installed and import it. * Make the module into a snippet to go into the ob-python.el. The first option is what the current patch does. The problem is that it requires users to install an external module. The second option avoids that but is uglier and harder to maintain. [1] There's a remaining issue, which I believe is specific to newer versions of python.el, where the first time a session block is executed, it doesn't wait for the shell to load up before sending input. >From 642370236f6ca06e70ea32866eedcdba161fd2c4 Mon Sep 17 00:00:00 2001 From: Kyle Meyer Date: Thu, 10 Dec 2015 22:09:05 -0500 Subject: [PATCH] WIP lisp/ob-python.el: Use block_eval --- lisp/ob-python.el | 108 +++--- 1 file changed, 54 insertions(+), 54 deletions(-) diff --git a/lisp/ob-python.el b/lisp/ob-python.el index 95a06f5..baaaf82 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -224,19 +224,18 @@ (defun org-babel-python-initiate-session (&optional session _params) (defvar org-babel-python-eoe-indicator "'org_babel_python_eoe'" "A string to indicate that evaluation has completed.") -(defconst org-babel-python-wrapper-method - " -def main(): -%s -open('%s', 'w').write( str(main()) )") +(defconst org-babel-python-block-eval-import + "from block_eval import block_eval as _ob_python_eval") + +(defconst org-babel-python-wrapper-method + (concat org-babel-python-block-eval-import " +open('%s', 'w').write(str(_ob_python_eval('%s')))")) (defconst org-babel-python-pp-wrapper-method - " + (concat org-babel-python-block-eval-import " import pprint -def main(): -%s - -open('%s', 'w').write( pprint.pformat(main()) )") +from block_eval import block_eval as _ob_python_eval +open('%s', 'w').write(pprint.pformat(str(_ob_python_eval('%s'")) (defun org-babel-python-evaluate (session body &optional result-type result-params preamble) @@ -247,6 +246,9 @@ (defun org-babel-python-evaluate (org-babel-python-evaluate-external-process
Re: [O] Proposal and RFC for improving ob-python
Thank you for the feedback Kyle. > As I understand it (and if I'm remembering correctly), there are at > least three issues with ob-python sessions: 4. session and non-session code differs significantly as you have already mentioned > So for non-session code and value results, the main idea is to get rid > of the need for "return" at the end? > Yes, that's the greatest offense to the Python code. > > This seems useful to me. Requiring a trailing "return", which is a > syntax error when the code is taken by itself, is a quirk of the > implementation (the code ends up being wrapped in a function to get the > return value). Also, it would allow switching from ":results value" to > ":results output" without needing to remove "return" in order to avoid a > syntax error. > Yes, these would be the obvious benefits. > But what about when :results != value? Doesn't your proposal only > handle returning the last value? > You mean :results output ? In that case it could just omit the "open(...).write(...) " part and capture anything the console prints before the primary prompt appears again. Or it could capture stdout into a file, but I think that's needlessly complicated: python -i << HEREDOC_END import sys sys.stdout = open() # replace stdout with some file _ = block_eval(""" """) sys.stdout.close() sys.stdout = sys.__stdout__ # restore stdout HEREDOC_END After thinking about this, perhaps it is better to capture the block result into "_" instead of "ret", because a) it prevents name clashes b) it is more in line with what "_" is used for Kind regards, Ondřej Grover
Re: [O] Proposal and RFC for improving ob-python
Ondřej Grover writes: [...] > The ob-ipython project tries to solve this hackiness in a different way by > using the client-server infrastructure of IPython/Jupyter. That works quite > well too, but my hope is that improving ob-python.el would also make it > simpler to use IPython as the python REPL, relying only on the core of the > Python language. As I understand it (and if I'm remembering correctly), there are at least three issues with ob-python sessions: 1. Shell prompt markers leak into the output. 2. Whitespace at the beginning of the output is swallowed. 3. Valid code that has indentation and spaces cannot be sent. I think the first one is the race conditions that ob-ipython.el's README refers to. > It essentially boils down to implementing progn-like eval() function in > Python which would return the result of the last statement if it is an > expression. [...] > My proposal is to implement an equivalent of the following bash pseudo code > for non session mode > > python -i << HEREDOC_END > ret = block_eval(""" > > """) > open().write(str(ret)) > HEREDOC_END So for non-session code and value results, the main idea is to get rid of the need for "return" at the end? This seems useful to me. Requiring a trailing "return", which is a syntax error when the code is taken by itself, is a quirk of the implementation (the code ends up being wrapped in a function to get the return value). Also, it would allow switching from ":results value" to ":results output" without needing to remove "return" in order to avoid a syntax error. > For session mode it would be even simpler, lines containing HEREDOC above > would be dropped and the rest piped directly into the Python REPL. > > This also means that the 'org_babel_python_eoe' string indicator may not be > necessary anymore because end of evaluation would be simply shown by a new > line with the primary prompt appearing. But what about when :results != value? Doesn't your proposal only handle returning the last value? -- Kyle
Re: [O] Proposal and RFC for improving ob-python
Hello, thank you all for confirming that I'm in the right place. If I felt confident enough that this would work, I wold have already presented a patch (I'm not an experienced ELisp programmer). That's why I want to discuss this approach first. Kyle, I'll be grateful for you feedback. Kind regards, Ondřej Grover On Mon, Dec 7, 2015 at 9:27 PM, Kyle Meyer wrote: > Hi Ondřej, > > Ondřej Grover writes: > > > TL;DR for those that may have been scared off by the length of my > previous > > email: > > I propose a method of improving ob-python.el by using a progn-like eval() > > Python function which can wrap and execute source blocks. > > > > If this list is not appropriate for discussion of improvements to > > ob-python.el, could you please give me pointers how to reach people that > > might be interested in helping and discussing? > > Thanks for your proposal. This is the right place to discuss it. I've > only had the chance to briefly look at this, but I'll try to take a > closer look soon. > > -- > Kyle >
Re: [O] Proposal and RFC for improving ob-python
Ondřej Grover writes: > If this list is not appropriate for discussion of improvements to > ob-python.el, could you please give me pointers how to reach people that > might be interested in helping and discussing? The best way to contribute to improving ob-python is by showing a patch that implements your ideas and still passes all existing tests. Bonus points if you add new tests for better coverage or any new functionality or improvements. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptation for Waldorf microQ V2.22R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Re: [O] Proposal and RFC for improving ob-python
Aloha Ondřej Grover, Ondřej Grover writes: > TL;DR for those that may have been scared off by the length of my previous > email: > I propose a method of improving ob-python.el by using a progn-like eval() > Python function which can wrap and execute source blocks. > > If this list is not appropriate for discussion of improvements to > ob-python.el, could you please give me pointers how to reach people that > might be interested in helping and discussing? > > Kind regards, > Ondřej Grover This is a good list to discuss improvements to ob-python. The authors of Babel in Org mode no longer participate in its development and AFAICT no one has taken over this responsibility. Instead, there have been changes to one language or another as issues arise. I do know that the Babel authors found ob-python difficult to write and that problems with it have been identified from time to time on this list. My sense is that it could be improved, though I don't use it often enough myself to know. At any rate, perhaps you could propose a patch and tests? If the patch breaks something, then you'll certainly find users interested in discussing ob-python :) All the best, Tom -- Thomas S. Dye http://www.tsdye.com
Re: [O] Proposal and RFC for improving ob-python
Hi Ondřej, Ondřej Grover writes: > TL;DR for those that may have been scared off by the length of my previous > email: > I propose a method of improving ob-python.el by using a progn-like eval() > Python function which can wrap and execute source blocks. > > If this list is not appropriate for discussion of improvements to > ob-python.el, could you please give me pointers how to reach people that > might be interested in helping and discussing? Thanks for your proposal. This is the right place to discuss it. I've only had the chance to briefly look at this, but I'll try to take a closer look soon. -- Kyle
Re: [O] Proposal and RFC for improving ob-python
TL;DR for those that may have been scared off by the length of my previous email: I propose a method of improving ob-python.el by using a progn-like eval() Python function which can wrap and execute source blocks. If this list is not appropriate for discussion of improvements to ob-python.el, could you please give me pointers how to reach people that might be interested in helping and discussing? Kind regards, Ondřej Grover On Sat, Dec 5, 2015 at 6:17 PM, Ondřej Grover wrote: > Hello, > > I've been playing around with the Org-mode Babel framework and I am > grateful to all the contributors for making this wonderful library. After > some time I noticed that Python support seems a little hacky and > inconsistent and after reading through ob-python.el and consulting Python > documentation I came up with a proposal for improving it. > > The ob-ipython project tries to solve this hackiness in a different way by > using the client-server infrastructure of IPython/Jupyter. That works quite > well too, but my hope is that improving ob-python.el would also make it > simpler to use IPython as the python REPL, relying only on the core of the > Python language. > > It essentially boils down to implementing progn-like eval() function in > Python which would return the result of the last statement if it is an > expression. I have come up with a prototype of such a function by diving > into Python scope internals and its AST capabilities. It was written using > Org-mode and tangling so it is thoroughly documented and explained, a test > suite is included. This interesting exercise made me appreciate Lisp even > more. Here it is > https://github.com/smartass101/python_block_eval > I haven't licensed it yet, because I'm not sure what license would be > appropriate if it was used by org-mode. Any suggestions? > > My proposal is to implement an equivalent of the following bash pseudo > code for non session mode > > python -i << HEREDOC_END > ret = block_eval(""" > > """) > open().write(str(ret)) > HEREDOC_END > > For session mode it would be even simpler, lines containing HEREDOC above > would be dropped and the rest piped directly into the Python REPL. > > This also means that the 'org_babel_python_eoe' string indicator may not > be necessary anymore because end of evaluation would be simply shown by a > new line with the primary prompt appearing. > > The reason why `python -i` (force interactive mode) is used is the IMHO > poor design choice in the CPython implementation (and other implementations > have a similar issue AFAIK) to have fast but RO access to local variable > scope in non-top-level (i.e. functions calling functions) frames/scopes. I > tried to hack my way around it in the update_locals_after_eval branch of my > repo to no avail, perhaps some Pythonista among you may know a solution. > > I also favor piping input into `python -i` because it means that a > temporary file does not have to be created. > > As explained in the README.org in my repo, this inconsistency can be > worked around by explicitly first evaluating the side-effect-only part of > the block and than the last expression with a direct eval() call for each. > This makes it longer by 2 lines, but has the advantage of properly handling > variable scope and separating side-effects, which could be used to e.g. > suppress output. Nevertheless, I think that for Org-mode Babel usage the > `python -i` and block_eval() approach suffices, unless someone finds a way > to use the advantages of the alternative approach to improve ob-python.el > even further. > > I'm not a skilled Elisp programmer, so I wanted to ask around as to the > feasibility of this endeavor or possibly availability of helping hands > before I devote more time to this. > > Kind regards, > Ondřej Grover >