Re: ENB: Cracking the Jupyter code

[vitalije]

| ​The script does not use getMessagesUntilIdle.  Does it do what you are 
talking about?​
This was remaining of some previous experiment which turned out to be 
unnecessary, but I have forgoten to delete it.

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[vitalije]

https://jupyter-client.readthedocs.io/en/stable/api/manager.html#utility-functions

This is where I have found it. I had no prior experience with jupyter at all. 
Vitalije

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[Edward K. Ream]

On Tue, Nov 7, 2017 at 10:24 AM, Terry Brown  wrote:

> it seems like non-blocking would be very valuable
> with computations on the Jupyter end that take a long time.


​ jupyter issue #242  asks for
help on exactly this topic.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[Edward K. Ream]

On Tuesday, November 7, 2017 at 10:02:12 AM UTC-6, Edward K. Ream wrote:

My next project will be to google to find out how to create objects that 
> use different values of traitlets defined as class members of jupyter 
> classes.
>

I have just filed jupyter issue #242 
 on the jupyter help page 
.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[Terry Brown]

Not sure where things are on the blocking/non-blocking front at the
moment, and not sure if I should keep opining with so little experience
of Jupyter :-), but it seems like non-blocking would be very valuable
with computations on the Jupyter end that take a long time.  So not
related to startup, but just cells that are unavoidably slow to execute
because of the amount of work required to process them.

Congrats. on the progress to date.

Cheers -Terry

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[Edward K. Ream]

On Tuesday, November 7, 2017 at 6:06:31 AM UTC-6, Edward K. Ream wrote:

Here is another approach.  Starting from your script, it caches the 
> kernel.  The second time it is run, it successfully uses the previously 
> created kernel.
>
> t1 = time.clock()
> d = g.app.permanentScriptDict
> kern = d.get('kernel')
> print('kernel...', kern)
> if not kern:
> # from run_kernel, which can't be used directly
> km = jupyter_client.KernelManager(kernel_name='python')
> km.start_kernel()
> kc = km.client()
> kc.start_channels()
> kc.wait_for_ready(timeout=60)
> d['kernel'] = kern = kc
>

 At last I see why the result from run_kernel can't be cached.  The code is:

@contextmanager
def run_kernel(**kwargs):
"""Context manager to create a kernel in a subprocess.

The kernel is shut down when the context exits.

Returns
---
kernel_client: connected KernelClient instance
"""
km, kc = start_new_kernel(**kwargs)
try:
yield kc
finally:
kc.stop_channels()
km.shutdown_kernel(now=True)

The @contextmanager decorator is part of Python's contextlib 
module, and defines a 
factory method for use with "with" statements.

As a result, run_kernel should be used within "with" statements.  Note that 
run_kernel is not documented on the api page 
, so I'm 
not sure how Vitalije intuited this.

Anyway, the following code *does* work (with getIn as before), using 
jupyter_client.manager.start_new_kernel instead of run_kernel.

t1 = time.clock()
d = g.app.permanentScriptDict
kern = d.get('kernel')
if kern:
print('existing kernel:', kern)
else:
print('creating kernel...')
junk, kern = jupyter_client.manager.start_new_kernel()
d['kernel'] = kern
kern.execute(code=code, user_expressions={'a':'a'})
msg = kern.get_shell_msg()
resultDict = getIn(msg, ['content', 'user_expressions', 'a'])
result = getIn(msg, ['content', 'user_expressions', 'a', 'data', 
'text/plain'])
g.printObj(resultDict)
print('%2.4f sec' % (time.clock()-t1))

This is much better than scooping out code from run_kernel because 
start_new_kernel will be updated as necessary when new releases come out.

My next project will be to google to find out how to create objects that 
use different values of traitlets defined as class members of jupyter 
classes.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: saving cloned @clean files in different @path's

[Phil]

Matt,
 

> I replicated the only-1-copy-written behaviour here with both @file and 
> @clean cloned nodes. 
>

Good! I'm not crazy...!


> I Edward explained that he really didn't like cloned 
> @file/@thin/@shadow/... nodes because of {...cache problems? race 
> conditions?} and possibility of data loss. (whee! I found it)
>
> The advised pattern is to have distinct `@file path1/thing` and `@file 
> path2/thing` head nodes and then create an organizer node inside each and 
> clone that instead.
>
 
I tried that, and indeed it works. However, it seems inconsistent that 
changing an organizer-node-clone would trigger file writes for all files 
containing that clone, but that changing an @file-node-clone would not do 
the same.

Further experimentation shows that only the *first* @file-node-clone that 
appears (in top-to-bottom traversal order) in the outline actually gets 
saved on change - subsequent clones are not saved to their respective 
paths. If I exchange the order of the clones in the outline, suddenly a 
different file is being saved, but none of the others.

If the reason for not using @file-node-clones is because of the risk of 
data loss, then my next question becomes "why is there a risk of data 
loss?" If the answer to this is due to some fundamental property of 
outlines containing clones, then I'd like to understand that better so that 
I can work accordingly. If that is not the case however, then the 
underlying cause of data loss should be addressed, rather than to simply 
adhere to a convention of not using @file-node-clones.

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: saving cloned @clean files in different @path's

[Matt Wilkie]

Hi Phil,

I replicated the only-1-copy-written behaviour here with both @file and
@clean cloned nodes.

I used to use cloned @file nodes in order to always have same copy of
`thing.py` in my development folder and in a shared system path. It saved
me from having to keep running a deploy script to update the copy. I really
liked that! Until... something weird happened and both copies got messed
up. And then a few weeks later it happened again. Edward explained that he
really didn't like cloned @file/@thin/@shadow/... nodes because of
{...cache problems? race conditions?} and possibility of data loss. (whee!
I found it)

The advised pattern is to have distinct `@file path1/thing` and `@file
path2/thing` head nodes and then create an organizer node inside each and
clone that instead.

This was a couple years ago. I don't know what might have changed since
then.

-Matt




On Mon, Nov 6, 2017 at 12:49 PM, Phil  wrote:

> Leo 5.6, build 20171105131700, Sun Nov  5 13:17:00 PST 2017
> Git repo info: branch = master, commit = 94ebac1f2510
> Python 2.7.13, PyQt version 5.6.2
> Windows 7 AMD64 (build 6.1.7601) SP1
> isPython3: False
> caching enabled
>
> Place the attached file in a folder containing empty subdirectories sub1
> and sub2. Then open the .leo file and do the following:
>
> 1) Select the test.c node under the sub1 path, make an edit to the body,
> and save the file. The file test.c will be created in directory sub1, but
> not in sub2.
>
> 2) Select the test.c node under the sub2 path, make an edit to the body,
> and save the file. The file test.c in directory sub1 will be modified, but
> there will be no file created in directory sub2.
>
> 3) With the test.c node under the sub2 path still selected, make an edit
> to the body, then save using Ctrl-Shift-W. The file test.c will be created
> in directory sub2 with the new edits, but the file in directory sub1 will
> not be updated.
>
> This behavior does not match what used to happen in the past (maybe more
> than a year ago). It used to be the case that changing a cloned @file-like
> node would result in those changes being saved to all the paths under which
> the clone lived in the Leo outline.
>
> I just recently switched to using the GIT repo as my Leo install, so maybe
> I lost a setting in that switch. Or maybe Leo's behavior changed. Either
> way, I'd like it to be what it used to be. Any suggestions?
>
> Thanks!
> Phil
>
> --
> You received this message because you are subscribed to the Google Groups
> "leo-editor" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to leo-editor+unsubscr...@googlegroups.com.
> To post to this group, send email to leo-editor@googlegroups.com.
> Visit this group at https://groups.google.com/group/leo-editor.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[Edward K. Ream]

On Tuesday, November 7, 2017 at 5:07:32 AM UTC-6, vitalije wrote:

Briefly, I looked at the code and I have just realized that my example 
> could be even simpler. Blocking client expects reply=True keyword argument, 
> when sending requests for computation and if it is found, than it waits for 
> the reply and returns it. 
>
> Look in jupyter_client.blocking.client module. There you can see what is 
> changed in blocking class.
>

It's mysterious to me how to subclass any of the jupyter classes.

Here is another approach.  Starting from your script, it caches the 
kernel.  The second time it is run, it successfully uses the previously 
created kernel.

t1 = time.clock()
d = g.app.permanentScriptDict
kern = d.get('kernel')
print('kernel...', kern)
if not kern:
# from run_kernel, which can't be used directly
km = jupyter_client.KernelManager(kernel_name='python')
km.start_kernel()
kc = km.client()
kc.start_channels()
kc.wait_for_ready(timeout=60)
d['kernel'] = kern = kc
kern.execute(code=code, user_expressions={'a':'a'})
msg = kern.get_shell_msg()
resultDict = getIn(msg, ['content', 'user_expressions', 'a'])
result = getIn(msg, ['content', 'user_expressions', 'a', 'data', 
'text/plain'])
print('%2.4f sec' % (time.clock()-t1))

The first time this is executed, the output is:

kernel... None
1.0270 sec

The second time, there is no need to create the kernel again:

kernel... 
0.0181 sec

In short, using a blocking client seems fast enough.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[vitalije]

I have no time to investigate deeper, but I would follow 
jupyter_client.run_kernel and look for place to change blocking client into 
unblocking one. 

Briefly, I looked at the code and I have just realized that my example 
could be even simpler. Blocking client expects reply=True keyword argument, 
when sending requests for computation and if it is found, than it waits for 
the reply and returns it. 

Look in jupyter_client.blocking.client module. There you can see what is 
changed in blocking class.

Now, I have to run. 
HTH Vitalije

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[Edward K. Ream]

On Tue, Nov 7, 2017 at 3:48 AM, Edward K. Ream  wrote:


On Tue, Nov 7, 2017 at 3:20 AM, Edward K. Ream  wrote:
>
> ​...​
>> Presumably there is a way to change client_class to:
>>
>> client_class = DottedObjectName('jupyter_client.KernelClient')
>>
>> but I don't know how to do this.
>>
>
> ​I'm wondering whether some **kwargs args may be used to do this.  This
> would be done implicitly via the traitlets mechanism.
>

​Still don't know.  Just on a hunch, I tried changing the Vitalije's script
to:

with jupyter_client.run_kernel(
client_class = 'jupyter_client.KernelClient',
kernel_name='python'
) as kern:​

But no joy.  kern is still a blocking client.  Closer reading of the code
shows that kwargs may affect the command being executed, but not the client
produced.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[Edward K. Ream]

On Mon, Nov 6, 2017 at 12:19 PM, vitalije  wrote:

Here is example script that I have made so far.
> with jupyter_client.run_kernel(kernel_name='python3') as kern:
>

​This will fail unless you are running python 3.

Using 'python' instead or 'python3' (or 'python2') will use the version of
python that you using.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[Edward K. Ream]

On Mon, Nov 6, 2017 at 12:19 PM, vitalije  wrote:

Here is example script that I have made so far.
> g.es(pprint.pformat(resultDict))
>

​Either of the following will print dicts and lists more clearly than
pprint.pformat:

g.printObj(resultDict)
g.es(g.objToString(resultDict))

I recommend them especially when dicts contain lists or other dicts.

The advantage of pprint is that it is guaranteed to be equivalent to the
original object.  That might be true of g.printObj and g.objToString, but
maybe not.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[Edward K. Ream]

On Tue, Nov 7, 2017 at 3:20 AM, Edward K. Ream  wrote:

​...​
> Presumably there is a way to change client_class to:
>
> client_class = DottedObjectName('jupyter_client.KernelClient')
>
> but I don't know how to do this.
>

​I'm wondering whether some **kwargs args may be used to do this.  This
would be done implicitly via the traitlets mechanism.

Reading the jupyter code is a horror show.  *Everything *is done implicitly.

Otoh, there may be an easy way to do what we want, if we can read between
the lines...

For example, the KernelManager class contains:

def _client_factory_default(self):
return import_item(self.client_class)

Here, import_item is defined in ipython_genutils/importstring.py.

Its docstring is:

 """Import and return ``bar`` given the string ``foo.bar``.

Calling ``bar = import_item("foo.bar")`` is the functional equivalent of
executing the code ``from foo import bar``.

Parameters
--
name : string
  The fully qualified name of the module/package being imported.

Returns
---
mod : module object
   The module that was imported.
"""

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[Edward K. Ream]

On Mon, Nov 6, 2017 at 12:19 PM, vitalije  wrote:

​> ​
Here is example script that I have made so far.

​Totally excellent!

​> ​
It doesn't do too much. It calculates sin(1).

​Actually, it shows how to evaluate any python string.​

Function jupyter_client.run_kernel returns a blocking version of jupyter
> client enclosed in a context. That means that the methods get_shell_msg,
> and others get_..._msg will block until the message arrives.
>

​The script does not use getMessagesUntilIdle.  Does it do what you are
talking about?​

​Clearly, I have not paid enough attention to the methods of the manager
module, to which jupyter_client.run_kernel refers.

This simplifies the example but for real use, we should use non-blocking
> client and read messages on iopub channel which inform us when kernel has
> started to execute our code (it becomes busy), and when it becomes idle
> again, we can read the result message from the shell channel.
>

​The KernelManager class contains these statements:

# the class to create with our `client` method
client_class =
DottedObjectName('jupyter_client.blocking.BlockingKernelClient')
client_factory = Type(klass='jupyter_client.KernelClient')

The Dotted and Type functions are part of the traitlets type system.
Presumably there is a way to change client_class to:

client_class = DottedObjectName('jupyter_client.KernelClient')

but I don't know how to do this.


> Instead of using every time another kernel enclosed in a python context
> which will expire when the context is left, perhaps kernel client can be
> created directly using jupyter_client.KernelClient(**kwargs) and instance
> stored in c.user_dict. It is non-blocking client.
>

​It's worth a try.

Many thanks for this script.  It's a big step forward.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Cracking the Jupyter code

[Edward K. Ream]

On Mon, Nov 6, 2017 at 7:21 PM, Offray Vladimir Luna Cárdenas <
off...@riseup.net> wrote:

>
>
> Maybe this could work:
>
> https://jupyter-client.readthedocs.io/en/latest/api/index.html


​In retrospect, the manager api is what I was looking for.  Thanks for this.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.