The design of Leo+Ipython+Jupyter+Lit-computing

[Edward K. Ream]

In this post, I'll discuss how Leo can integrate with IPython and Jupyter 
notebooks so as to become a premier literate *computing *environment, as 
Offray uses the term.

This is a design document.  I'll focus only on what the user sees and 
experiences, not on how Leo might actually get the job done.* tl:dr:*  Read 
the summary.

*IPython and Jupyter will remain essentially unchanged*

I assume that both IPython and Jupyter will not change in their essential 
qualities.  That is, IPython and Jupyter are *fixed* (constant).  In 
particular, I assume that for the foreseeable future that Jupyter notebooks 
will be a *linear* list of cells, not a tree of cells. In any case, the 
following design would not change much if Jupyter notes ever *do* support 
trees of cells.

*Implications*

1. As a direct consequence of this assumption, only Leo will explicitly 
deal with outlines.

2. There must be some way of converting (parts of) Leo outlines to a *list* 
of .ipynb files. There is considerable room for creativity in this 
conversion process.  That is, we want to keep tree-to-notebooks process in 
the back of our mind as we think about Leo trees.

*The Jupyter design*

The jupyter notebook handles cells in a simple, intuitive manner as follows:

1. Running a code cell produces output *in that cell*.  That is, the output 
is attached to the cell itself, not in a Qt popup window.

2. It's possible to switch the types of Jupyter cells from a dropdown menu. 
The cell types are Code, Markdown, Raw NB Convert and Heading.

3. There are Jupyter commands to run some or all cells, and to insert, 
delete and move cells.

*A preliminary design*

Imo, we probably want similar (or improved) behavior in Leo cells:

- It should not be necessary to open a view-rendered window to see the 
output of a cell.

- Cells should show the In and Out numbers, just like in IPython or Jupyter.

- Leo directives will indicate the type of cells.  Some of these directives 
will apply to descendant nodes.

- Leo will provide commands to run all cells, selected cells, selected 
subtrees and marked cells.

- Existing Leo commands will suffice to insert, delete, move, clone, 
promote and demote cells.

*New directives*

*@jupyter-notebook*: explicitly denotes a notebook.  All descendants are 
IPython cells. Useful when converting a Leo outline to one or more Jupyter 
notebooks.

*@cells*: All descendants are IPython cells.

*@cell*: A single IPython cell.

Within a cell, @language denotes the type of cell.

@language markdown denotes a Jupyter Markdown cell.  There probably is not 
great need for Jupyter Heading cells: they may come free with Leo's outline 
structure.

@language python (or julia, or r, etc) denotes a code cell.

*Behavior*

There is lots of room for experimentation re behavior and rendering.  In 
general, we want cells to work just like the corresponding Jupyter cells, 
but enhanced, when possible, by Leo's outline structure.

For example, the rending of @markdown cells might include the rendering of 
all descendant @markdown cells, thereby making Jupyter Heading cells 
unnecessary with Leo itself.  Otoh, Leo will "reconstitute" Heading cells 
when exporting @jupyter-notebook trees.


*Summary*
1. New Leo directives will designate subtrees as containing IPython *cells*. 
Such cells will look and work like individual IPython cells.  They will 
show In and Out numbers, and will show output directly, without the need 
for a viewrendered pane.

2. Leo's existing @language directives will indicate the type of cell. Such 
directives could affect which Jupyter kernel evaluates Code nodes.

3. Wherever possible, Jupyter commands and cells will be enhanced by Leo's 
outline structure. For example, the rendering of an @markdown tree may make 
Jupyter Heading cells unnecessary.

4. Converting an outline to one or more .ipynb files might allow easy 
cross-file references in the resulting Jupyter notebooks.  A 
table-of-contents notebook would likely be easy to create.

All comments welcome.

Edward

P. S. To prepare for this post, I have carefully read all the comments and 
references in the Leo + PlantUML workflow 
 thread. 
Naturally, I may have misunderstood something. Please correct me if so.

EKR

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Understanding Leo Code

[Offray Vladimir Luna Cárdenas]

Hi,


On 09/01/17 15:36, Edward K. Ream wrote:
On Mon, Jan 9, 2017 at 7:36 AM, rengel > wrote:


This might be of interest to anyone who tries to understand and
program Leo code.

http://www.laputan.org/mud/mud.html


It also might help to understand why it seems so difficult to find
more Leo adopters.


​Hmm.  Leo is not a big ball of mud.  It's architecture has remained 
almost completely unchanged for decades.  But the plumbing and wiring 
change continually ;-)


Leo's problems attracting users stem from other sources. Probably the 
most important is the need for .leo files.  In other words, people are 
unwilling to pay even small prices for better tools.


Edward


But I think that .leo files (or the ideas behind it) are a bless for 
literate computing (as explained in detail in my other long post). A lot 
of user could come from there.


Cheers,

Offray

--
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Understanding Leo Code

[Edward K. Ream]

On Mon, Jan 9, 2017 at 7:36 AM, rengel  wrote:

> This might be of interest to anyone who tries to understand and program
> Leo code.
>
> http://www.laputan.org/mud/mud.html
>
> It also might help to understand why it seems so difficult to find more
> Leo adopters.
>

​Hmm.  Leo is not a big ball of mud.  It's architecture has remained almost
completely unchanged for decades.  But the plumbing and wiring change
continually ;-)

Leo's problems attracting users stem from other sources. Probably the most
important is the need for .leo files.  In other words, people are unwilling
to pay even small prices for better tools.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Understanding Leo Code

[rengel]

This might be of interest to anyone who tries to understand and program Leo 
code.

http://www.laputan.org/mud/mud.html

It also might help to understand why it seems so difficult to find more Leo 
adopters.


-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Switching focus to Leo via Alt-TAB sometimes very slow (W7)?

[jkn]

a bit of extra information, in case it helps:

- I have a two-monitor system on my W7 desktop system (only)
- I have most Aero-type effects turned off; Alt-TABbing just shows the bar 
in the middle of the monitor, with small icons that I step between. No 
fancy window previews or anything like that...

Thanks
Jon N

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Switching focus to Leo via Alt-TAB sometimes very slow (W7)?

[jkn]

I have just remembered to ask about this:

I run Leo mainly on a W7 desktop, and also occasionally on a couple of 
Linux laptops and desktops.

On the W7 machine, I notice that sometimes, when I ALT-TAB to Leo, there is 
a substantial delay (1 to several seconds) after releasing the ALT-TAB, and 
the Leo window being fully painted.

I am pretty sure that I have not seen this on the the Linux systems, 
although I use Leo less often on those.

The log pane shows:

{{{
Leo 5.1-final, build 20160201145421, Mon Feb  1 14:54:21 CST 2016
Not running from a git repo
Python 2.7.2, PyQt version 4.8.6
Windows 7 x86 (build 6.1.7601) SP1
}}}

but I am pretty sure I have seen this for quite a long time.

I mention this as it gives an unfortunate 'cognitive load' to the thought 
'switch to Leo and do X ...'. FWIW, once the main window has the focus all 
seems OK.

Has anyone else seen anything like this? I would be happy to learn of ways 
of fixing it...

Thanks
Jon N

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Embedding Leo into pyzo?

[Satish Goda]

I think in today's world of disparate software components, It would be 
really nice to see how Leo's integration component would integrate with 
Pyzo.

I really wish for a future where plug and play of awesome components from 
various disciplines will be as easy and painless as possible.

On Saturday, January 7, 2017 at 9:33:07 PM UTC+8, Edward K. Ream wrote:
>
> I've been thinking of this ever since Satish Goda's original PlantUML and 
> pyzo posts.
>
> Yes, it would theoretically be possible to embed Leo into pyzo, but the 
> sane, sensible (and polite) approach is to leave pyzo alone, and make Leo 
> do everything that pyzo can do ;-)
>
> Otoh, for study only, it may be useful to insert traces into the actual 
> pyzo editor.  These will never become public.
>
> Edward
>

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Improving Leo using pyzo code

[Satish Goda]

Hello Edward.

Really nice to see that you are moving forward with this. I am very happy 
and glad that I referred Pyzo, although I was a bit reluctant to mention it 
with the assumption that it would disrupt the Leo/PlantUml discussion. 

One use case in Pyzo that I really used a lot in my previous movie project 
was the multiple interpreters feature in Pyzo. After looking at Leo, Leo 
could support multiple interpreters in the same session be defining them 
across two trees in the same outline. The Top level could define the 
interpreter settings etc and nodes below that could be cells.


On Saturday, January 7, 2017 at 9:21:19 PM UTC+8, Edward K. Ream wrote:
>
> Good Artists Copy; Great Artists Steal 
> --unknown/various
>
> tl;dr: pyzo is the work of a programming genius.  I'll use that genius to 
> improve Leo.
>
> Pyzo contains absolutely brilliant code that I shall be "borrowing" 
> freely.  The pyzo code is covered by the new BSD license, so that will have 
> to be reflected in various places.  No big deal.
>
> After studying pyzo briefly, four or five areas for improving Leo present 
> themselves:
>
>
> *ipython integration*
> pyzo defines and uses the yoton client/server library.  Leo will likely 
> use yoton just like pyzo does. This will be a major project, but imo it is 
> the future of Leo. It will make Leo a first-class IPython/Jupyter editor, 
> just as pyzo is. This project should reduce or eliminate the need for Leo's 
> IPython bridge. 
>
> *syntax coloring*
>
> pyzo's syntax coloring looks to be more than 10x faster than Leo's.  And 
> it's so much simpler. Leo simply must use this scheme.  It will take 
> considerable adapter code.
>
> In particular, pyzo uses brilliantly simple token-based parsers to guide 
> syntax coloring. These parsers use regex's to generate tokens.  At present, 
> pyzo can only colorize python and c. Leo will have to generate regex's for 
> each language using files in leo/modes.
>
> This project is worth any amount of work. It should allow us to remove the 
> odious "big text" buttons.
>
> *autocompletion*
>
> pyzo's autocompletion is simpler, faster and just plain better than Leo's. 
> I'm not sure yet whether it depends on ipython, but I'll use as much of the 
> pyzo code as possible.
>
> *key handling*
>
> Parts of Leo's key handling code are scandalously complex.  The pyzo code 
> is much simpler.  I'll check whether the front end of Leo's key handling 
> code can be simplified using the pyzo code as a model.
>
>
> *Tree handling*
> pyzo's tree handling code is super simple, but that's because the pyzo 
> code doesn't need to do much. In particular, the pyzo code need only 
> respond to mouse clicks and standard keys--it doesn't have to deal with 
> events generated by custom keys or python scripts.  Still, the code will be 
> worth a close look, to see if Leo's code could be improved.
>
> Edward
>

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Leo + PlantUML workflow

[Satish Goda]

On Saturday, January 7, 2017 at 1:17:16 AM UTC+8, Edward K. Ream wrote:
>
> ​On Thu, Jan 5, 2017 at 8:48 PM, 
> ​​
> Satish Goda  wrote:
>
>> ​​I have used Jupyter notebooks sparingly, but I do like their utility 
>> and interactivity.
>>
>> At my workplace last year, I have successfully integrated Pyzo in my team 
>> for interactive python debugging and testing workflows.
>>
>
> ​I am studying Pyzo right now.  I am super impressed with their python 
> parser and syntax highlighter. The code is brilliantly simple.  Leo has to 
> use this approach. It's much, much faster than Leo's.
>
> Cells are also cool, as is IPython integration.   It should be easy to 
> have Pyzo recognize Leo nodes as cells.  Just treat #+node like ##.
>
> Pyzo is a serious competitor to Leo, and is also an inspiration for Leo.  
> Indeed, Leo can now longer settle for wimpy IPython integration.  Leo must 
> have an IPython pane, as in Pyzo.  This requires heavy lifting, with yoton 
> , kernels, etc. The details are 
> complex, and I don't understand them yet.
>
> ​> ​
> As you said, a Leo @jupyter node's children could be executable cells!
>
> ​There is a lot going on in my mind about this.  It's not entirely clear 
> what Leo's new gui is going to be like.  This is the important thing at the 
> design level.  If Leo is going to supplement either IPython or Pyzo, it 
> must, in some sense, be as easy to use as they are.
>
> The technical side is also going to be interesting/challenging.  I'm not 
> sure how to proceed. One possibility would be to add some prototyping code 
> to Pyzo.  Or vice versa: add some prototyping code to Leo.
>
> In my mind, this is a super important project.  For me, Leo must support 
> IPython and Jupyter.  Otherwise, Leo can never be main stream.  As a 
> result, this project is worth any amount of work.
>
> Naturally, I won't just start hacking away on code (except for minor 
> prototypes).  The main goal is to support Jupyter notebooks as well as 
> humanly possible.  This goal isn't about code, it's about design.
>
> Many thanks, ​Satish, for this post.  It seems to have come at the perfect 
> time to inspire me, both at the code level and the design level.
>  
>

> All comments welcome, Amigos.
>
> Edward
>


My pleasure Edward. I am exclusively using Leo since two months (not even 
Pyzo/PyCharm) as I am able to document my thoughts and move things around 
easily (non destructively via clone nodes). The executing of code and its 
output is something we could address in 2017 with a revised code/execution 
architecture.

I am also studying thoroughly jupyter/yoton and leo and hopefully I can 
bring more ideas to the table. 

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.