Re: ENB: Cracking the Jupyter code
Hi, On 06/11/17 10:56, Edward K. Ream wrote: > The problem we all have, I think, is that there appears to be no "how > to" documents regarding jupyter clients, servers, kernels, sessions, > etc. Please let me know if I have missed something. Maybe this could work: https://jupyter-client.readthedocs.io/en/latest/api/index.html Cheers, Offray -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
saving cloned @clean files in different @path's
Leo 5.6, build 20171105131700, Sun Nov 5 13:17:00 PST 2017 Git repo info: branch = master, commit = 94ebac1f2510 Python 2.7.13, PyQt version 5.6.2 Windows 7 AMD64 (build 6.1.7601) SP1 isPython3: False caching enabled Place the attached file in a folder containing empty subdirectories sub1 and sub2. Then open the .leo file and do the following: 1) Select the test.c node under the sub1 path, make an edit to the body, and save the file. The file test.c will be created in directory sub1, but not in sub2. 2) Select the test.c node under the sub2 path, make an edit to the body, and save the file. The file test.c in directory sub1 will be modified, but there will be no file created in directory sub2. 3) With the test.c node under the sub2 path still selected, make an edit to the body, then save using Ctrl-Shift-W. The file test.c will be created in directory sub2 with the new edits, but the file in directory sub1 will not be updated. This behavior does not match what used to happen in the past (maybe more than a year ago). It used to be the case that changing a cloned @file-like node would result in those changes being saved to all the paths under which the clone lived in the Leo outline. I just recently switched to using the GIT repo as my Leo install, so maybe I lost a setting in that switch. Or maybe Leo's behavior changed. Either way, I'd like it to be what it used to be. Any suggestions? Thanks! Phil -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout. test.leo Description: Binary data
Re: ENB: Cracking the Jupyter code
Here is example script that I have made so far. import jupyter_client import pprint code = '''import math a = math.sin(1)''' def getIn(r, pth): for p in pth: r = r.get(p, {}) return r def getMessagesUntilIdle(kern): resp = [] while True: msg = kern.get_iopub_msg() estate = getIn(msg, ['content', 'execution_state']) resp.append(msg) if estate == 'idle': break return resp with jupyter_client.run_kernel(kernel_name='python3') as kern: kern.execute(code=code, user_expressions={'a':'a'}) # after executing given code, kernel replies with the message # containing the expressions we have asked (i.e. 'a') msg = kern.get_shell_msg() # each expression we asked for is returned as a dict resultDict = getIn(msg, ['content', 'user_expressions', 'a']) # result as plain text result = getIn(msg, ['content', 'user_expressions', 'a', 'data', 'text/plain']) g.es('-code') g.es(code) g.es('---result:', result) g.es(pprint.pformat(resultDict)) It doesn't do too much. It calculates sin(1). Function jupyter_client.run_kernel returns a blocking version of jupyter client enclosed in a context. That means that the methods get_shell_msg, and others get_..._msg will block until the message arrives. This simplifies the example but for real use, we should use non-blocking client and read messages on iopub channel which inform us when kernel has started to execute our code (it becomes busy), and when it becomes idle again, we can read the result message from the shell channel. Instead of using every time another kernel enclosed in a python context which will expire when the context is left, perhaps kernel client can be created directly using jupyter_client.KernelClient(**kwargs) and instance stored in c.user_dict. It is non-blocking client. HTH Vitalije -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: Cracking the Jupyter code
On Mon, 6 Nov 2017 07:03:28 -0800 (PST) vitalijewrote: > > It is now clear that it's possible to create kernels without > > interacting with the Jupyter notebook server. In retrospect, Terry > > has likely been trying to tell me this for awhile... > > I believe it is quite opposite of what Terry has suggested. If I > understood well, Terry was suggesting interacting (sending and > receiving messages) with server, and leave kernels alone. Kernels > should be server's concern not Leo's. Server will create and deal > with them upon receiving message from Leo. Leo should not create > directly kernels. Otherwise, Leo will need to re-implement all code > from server that is intended to deal with kernels. > > Terry, please correct me if I misunderstood you. > Vitalije I think in general Vitalije's correct about the approach I was trying to advocate. I know at one point I was conflating the terms kernel and server, which might have contributed to confusion. I guess the component I should have been referring to is the server, not the kernel - I had noted that they might not be equivalent. I've tried to stress that Jupyter itself being written in Python should be irrelevant - a "service" that executes Python in a rich environment (as Jupyter does) is attractive to (some subset of) Leo's Python users, regardless of how the service is implemented. For example here's a web service: http://www.melissadata.com/lookups/latlngzip4.asp?lat=45=-92 that converts lat and lon to zip code and some HTML showing a map I'm hoping we can treat Jupyter as a service like that, converting Python code to results (JSON) and maybe HTML displaying that result. Cheers -Terry -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: Cracking the Jupyter code
On Mon, Nov 6, 2017 at 9:03 AM, vitalijewrote: > It is now clear that it's possible to create kernels without interacting >> with the Jupyter notebook server. In retrospect, Terry has likely been >> trying to tell me this for awhile... >> > > I believe it is quite opposite of what Terry has suggested. If I > understood well, Terry was suggesting interacting (sending and receiving > messages) with server, and leave kernels alone. Kernels should be server's > concern not Leo's. Server will create and deal with them upon receiving > message from Leo. Leo should not create directly kernels. Otherwise, Leo > will need to re-implement all code from server that is intended to deal > with kernels. > The problem we all have, I think, is that there appears to be no "how to" documents regarding jupyter clients, servers, kernels, sessions, etc. Please let me know if I have missed something. I have spent considerable time looking at the existing jupyter_client demo (__main__.py). I can see how the code handles the messages, but the init process is insanely complicated. I have no idea how to reuse the code. The jupyter_client demo creates a client, several sessions, and at least one kernel. Afaik, it does not create a stand-alone server, but I could be wrong. As I have said before, this project borders on being too complex for me to understand. I certainly do not understand how this could work with Leo. In my talk this morning with my friend Phil Straus, I said it is time for me to put this project on hold. There are many other tasks that I have neglected recently, and it's hard for me to justify this kind of speculative work. I would welcome any help you, Terry, or anyone else could give. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: Cracking the Jupyter code
> > It is now clear that it's possible to create kernels without interacting > with the Jupyter notebook server. In retrospect, Terry has likely been > trying to tell me this for awhile... > I believe it is quite opposite of what Terry has suggested. If I understood well, Terry was suggesting interacting (sending and receiving messages) with server, and leave kernels alone. Kernels should be server's concern not Leo's. Server will create and deal with them upon receiving message from Leo. Leo should not create directly kernels. Otherwise, Leo will need to re-implement all code from server that is intended to deal with kernels. Terry, please correct me if I misunderstood you. Vitalije -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: Cracking the Jupyter code
What I meant to say is that there is no need for Qt application to run in same process as Leo. In essence you are (or should be) trying to replace jupyter_console with Leo. So, if you use any code from jupyter_console, what than is left for Leo to do? If you like you can investigate how jupyter_console works, but not instantiate or run any part of it in Leo process. I am totally guessing this, I haven't done anything with jupyter yet. This is what I believe according to the discussion so far. Vitalije -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: Cracking the Jupyter code
I am following discussions about Leo - Jupyter integration, although I have no time to actually try or do anything about it. Edward, I suspect that you didn't understand very well what Terry has suggested on several occasions. There is no need to instantiate kernels or any part of the server side code from Leo. It would suffice for Leo just to: 1. check if server is already running and if not to execute an external script which should run server in separate process. 2. After that Leo scripts should just create messages and send them to server through zeromq, 3. and wait for the response messages again through zeromq connections. Everything is in formatting messages adequately using data from Leo tree and parsing/unpacking data from responses when they arrive displaying somehow in Leo. HTH Vitalije -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: Cracking the Jupyter code
On Monday, November 6, 2017 at 8:10:01 AM UTC-6, Edward K. Ream wrote: The solution may be to ignore "packaging" (existing classes) and just focus > on these tasks: > > - Create and init a kernel. > - Send and receive an "execute" message. > > It's not clear that *any* jupyter code can be used, except imports... > That last statement may be too pessimistic. It ignores several mixin classes, such as KernelClient, a subclass of ConnectionFileMixin and possibly BaseFrontendMixin and QtKernelManagerMixin. These probably can be instantiated without too much trouble. And if they can't be instantiated easily, they can be "distilled" into new classes... Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: Cracking the Jupyter code
On Monday, November 6, 2017 at 7:06:27 AM UTC-6, Edward K. Ream wrote: But for now I'm enjoying learning as much as I can about the jupyter code. > So I'll follow the high-energy path and not worry about where it leads... > I'm beginning to doubt whether the code can be distilled as I first envisioned. For example, the following fails when run from Leo: # from jupyter_console.__main__.py from jupyter_console import app app.launch_new_instance() This code gives the message endlessly when run from Leo: QCoreApplication::exec: The event loop is already running Apparently, there is Qt code in the jupyter console app. It's extremely messy. The alternative would be to perform a more radical distillation of the code, trying to create a working analog of Terry's "totally made up" pseudo code, namely: connection = zeromq.connect("http://127.0.0.1:8901/notebooks/book2;) cell_id = connection.add_cell(type='code') connection.set_content(cell_id, "import math\na=math.sin(1)") connection.run_cell(cell_id) html = connection.get_html(cell_id) There are too many details to keep straight in my mind. My previous, not-really-working code was: def make_channel(channel_name): print('-'*20) put('channel_name', channel_name) url = cl._make_url(channel_name) put('url', url) cl.log.debug("connecting %s channel to %s" % (channel_name, url)) socket = cl.connect_shell(identity=cl.session.bsession) put('socket', socket) channel = ZMQSocketChannel(socket, cl.session) # New code: don't use the various _x_channel_class values. put('channel', channel) return channel put('g.isPython3', g.isPython3) cl = client.KernelClient() put('shell_channel_class', cl.shell_channel_class) put('session', cl.session) cl._shell_channel = make_channel('shell') cl._iopub_channel = make_channel('iopub') cl._stdin_channel = make_channel('stdin') cl._hb_channel = make_channel('hb') cl.allow_stdin = True *Summary* I am at my limit of my working memory, trying to keep all the various approaches in my head. The "distilling" project is foundering during initialization, which is mind-boggling. The "prototype" project is foundering because of it's incompleteness. The solution may be to ignore "packaging" (existing classes) and just focus on these tasks: - Create and init a kernel. - Send and receive an "execute" message. It's not clear that *any* jupyter code can be used, except imports... Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: Cracking the Jupyter code
On Monday, November 6, 2017 at 5:46:05 AM UTC-6, Edward K. Ream wrote: Mystery solved. Results are published on the io_pub channel. New traces > show that ZMQTerminalInteractiveShell.handle_iopub contains a switch on the > message type: > This ZMQTerminalInteractiveShell code is different (more complex) from the corresponding Leonine code because it deals with user interactions. Otoh, distilling existing code may not provide all the code we need. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: Cracking the Jupyter code
On Monday, November 6, 2017 at 4:44:01 AM UTC-6, Edward K. Ream wrote: *Oops*. "4" appears in Out[1] *before* the call to handle_execute_reply. > In any case, the reply message does not contain "4" anywhere. Clearly, > more investigation is required. > Mystery solved. Results are published on the io_pub channel. New traces show that ZMQTerminalInteractiveShell.handle_iopub contains a switch on the message type: In [1]: 777 execute (KernelClient) code: 777 execute = msg sent handle_iopub (ZMQTerminalInteractiveShell) a3e1745c-5f58-4b30-844c-b527e2e381b1 handle_iopub msg_type status handle_iopub msg_type status handle_iopub msg_type status handle_iopub msg_type status handle_iopub msg_type status handle_iopub msg_type status handle_iopub msg_type status handle_iopub msg_type status handle_iopub msg_type status handle_iopub msg_type status handle_iopub msg_type status handle_iopub msg_type execute_input handle_iopub msg_type execute_result Out[1]: 777 handle_iopub Done handle_iopub msg_type status handle_execute_reply (ZMQTerminalInteractiveShell) handle_iopub (ZMQTerminalInteractiveShell) a3e1745c-5f58-4b30-844c-b527e2e381b1 Here is the relevant code: def handle_iopub(self, msg_id=''): """Process messages on the IOPub channel This method consumes and processes messages on the IOPub channel, such as stdout, stderr, execute_result and status. It only displays output that is caused by this session. """ g.trace('(ZMQTerminalInteractiveShell)', msg_id) while self.client.iopub_channel.msg_ready(): sub_msg = self.client.iopub_channel.get_msg() msg_type = sub_msg['header']['msg_type'] g.trace('msg_type', msg_type) parent = sub_msg["parent_header"] if self.include_output(sub_msg): if msg_type == 'status': self._execution_state = sub_msg["content"]["execution_state"] ... elif msg_type == 'execute_result': if self._pending_clearoutput: print("\r", file=io.stdout, end="") self._pending_clearoutput = False self.execution_count = int(sub_msg["content"]["execution_count"]) if not self.from_here(sub_msg): sys.stdout.write(self.other_output_prefix) format_dict = sub_msg["content"]["data"] self.handle_rich_data(format_dict) # taken from DisplayHook.__call__: hook = self.displayhook hook.start_displayhook() hook.write_output_prompt() # EKR: Writes Out[...] hook.write_format_data(format_dict) hook.log_output(format_dict) hook.finish_displayhook() g.trace('Done') As you can see, there are complications. In general, we want to init everything so that we can use the code above unchanged. We shall see how easy it will be to have the init code "do all the work". Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: Cracking the Jupyter code
On Monday, November 6, 2017 at 4:26:13 AM UTC-6, Edward K. Ream wrote: > > *Tracing messages* > > I also added, again by trial and error, traces show how messages are sent > and replies received. Typing "2+2" into the input cell results in: > > get_msg (ZMQSocketChannel block True timeout 1.0 > execute (KernelClient) code: 2+2 > get_msg (ZMQSocketChannel block True timeout 0.05 > get_msg (ZMQSocketChannel block True timeout None > get_msg (ZMQSocketChannel block True timeout None > get_msg (ZMQSocketChannel block True timeout None > get_msg (ZMQSocketChannel block True timeout None > get_msg (ZMQSocketChannel block True timeout None > get_msg (ZMQSocketChannel block True timeout None > get_msg (ZMQSocketChannel block True timeout None > get_msg (ZMQSocketChannel block True timeout None > get_msg (ZMQSocketChannel block True timeout None > Out[1]: 4 > get_msg (ZMQSocketChannel block True timeout None > get_msg (ZMQSocketChannel block False timeout 0.05 > handle_execute_reply (ZMQTerminalInteractiveShell) > [snip] *Oops*. "4" appears in Out[1] *before* the call to handle_execute_reply. In any case, the reply message does not contain "4" anywhere. Clearly, more investigation is required. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: Cracking the Jupyter code
On Monday, November 6, 2017 at 4:26:13 AM UTC-6, Edward K. Ream wrote: > Apparently, the app only uses a single session for communicating with the kernel. As I re-read the traces, I am not so sure about this. Probably not a big deal. The distillation process will reveal what is going on more clearly. EKR -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
ENB: Cracking the Jupyter code
This is an Engineering Notebook post. Feel free to ignore. Otoh, this very long post documents several major milestones. At last I have a simple, effective method for exploring jupyter code that involves reading neither code nor docs ;-) Of course, I've done enough reading of both by now to have some background... The technique is to insert g.trace statements that show how the *installed *jupyter packages work. These traces appear in the jupyter_client, jupyter, jupyter_core and qtconsole packages. The rest of this post explains what is happening in detail, editing out all the meanderings involved in the discovery process. *Starting the demo app* The following starts the console-based demo: cd C:\Anaconda2\Lib\site-packages\jupyter_console python2 __main__.py This starts qt-based app: cd C:\Anaconda2\Lib\site-packages\qtconsole python2 __main__.py At last I see that using the console demo is simpler because there is no qt code involved. *Preparing to trace* I added the following lines to all files (in the python2 library) containing code I wanted to trace: import leo.core.leoGlobals as g assert g I was unsure whether stdout was being redirected as in server code. Would I need to use logging messages? To be absolutely sure I would know when code was being executed, I added calls to g.pdb(). It turns out that g.trace *does* work as always, which greatly simplifies matters. *Tracing init code* At first, it was difficult to determine what code is being executed. There are lots of similar classes involved. To disambiguate the traces, I added explicit indications of what class was involved. Like this, for the Session ctor: g.trace('(Session)', kwargs) And here is the result of starting the app: >c:\Anaconda2\python.exe __main__.py C:\Anaconda2\Lib\site-packages\jupyter_console>c:\Anaconda2\python.exe __main__.py init_kernel_manager (JupyterConsoleApp) __init__ (Session) {'parent': } init_kernel_client (JupyterConsoleApp) get_connection_info (ConnectionFileMixin) { control_port: 53176, hb_port: 53177, iopub_port: 53174, ip: u'127.0.0.1', session: , shell_port: 53173, stdin_port: 53175, transport: u'tcp' } start_channels (KernelClient) get_msg (ZMQSocketChannel block True timeout 1 __init__ (Session) {'parent': } get_msg (ZMQSocketChannel block True timeout 1 Jupyter Console 4.1.1 get_msg (ZMQSocketChannel block True timeout 1 get_msg (ZMQSocketChannel block True timeout 1 In [1]: The last line is the typical ipython/jupyter cell prompt. *Executing commands* A vital step was figuring out what code actually handles the key activities of the app. After much mucking about, I realized that the "execute" method handles code execution. There are also "history", "completion" and "inspection" methods. For the console app, these methods reside in the KernelClient class.* These methods are the keys that unlock the jupyter code*. Leo will use these methods, perhaps slightly modified, when implementing cells within Leo. *Tracing messages* I also added, again by trial and error, traces show how messages are sent and replies received. Typing "2+2" into the input cell results in: get_msg (ZMQSocketChannel block True timeout 1.0 execute (KernelClient) code: 2+2 get_msg (ZMQSocketChannel block True timeout 0.05 get_msg (ZMQSocketChannel block True timeout None get_msg (ZMQSocketChannel block True timeout None get_msg (ZMQSocketChannel block True timeout None get_msg (ZMQSocketChannel block True timeout None get_msg (ZMQSocketChannel block True timeout None get_msg (ZMQSocketChannel block True timeout None get_msg (ZMQSocketChannel block True timeout None get_msg (ZMQSocketChannel block True timeout None get_msg (ZMQSocketChannel block True timeout None Out[1]: 4 get_msg (ZMQSocketChannel block True timeout None get_msg (ZMQSocketChannel block False timeout 0.05 handle_execute_reply (ZMQTerminalInteractiveShell) { buffers: [], content: { execution_count: 1, payload: [], status: u'ok', user_expressions: {} }, header: { date: datetime.datetime(2017, 11, 6, 3, 31, 54, 429000), msg_id: u'f59b9f4c-ca27-4630-a138-5ee0e3b0a2a6', msg_type: u'execute_reply', session: u'187e92d1-9cbe-4126-ad32-774e371cd1d6', username: u'username', version: u'5.0' }, metadata: { dependencies_met: True, engine: u'f2e67b09-71fd-4e00-a60b-4c556593ebc7', started: u'2017-11-06T03:31:54.413000', status: u'ok' }, msg_id: u'f59b9f4c-ca27-4630-a138-5ee0e3b0a2a6', msg_type: u'execute_reply', parent_header: { date: datetime.datetime(2017, 11, 6, 3, 31, 54, 413000), msg_id: u'6035ad0f-592e-42bc-bc4d-3df628073815', msg_type: u'execute_request', session: u'b982860d-e2e2-414b-8562-f9a0c6ee7d73', username: u'username', version: u'5.0' } } *Waiting for replies* The traces