Re: Syntax coloring mystery solved!!

['Terry Brown' via leo-editor]

On Sun, 15 Jan 2017 06:18:53 -0500
"Edward K. Ream"  wrote:

> Terry, please test this branch for a day or so and report any
> problems. I'll follow this with interest on my travels. If all seem
> well, feel free to merge the branch.

Initial look I'm not seeing anything but way faster colorizing :-)
Will report if anything else shows up.

Cheers -Terry

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Syntax coloring mystery solved!!

[Edward K. Ream]

On Sat, Jan 14, 2017 at 11:52 PM, 'Terry Brown' via leo-editor <
leo-editor@googlegroups.com> wrote:

Branch is there in the correct place.
>

​Thanks for the confirmation.​


​Rev 0a2a983 fixes all known colorizer bugs, and rev fa3d41e is a bit of
make-work while eating my own dog food.

Terry, please test this branch for a day or so and report any problems.
I'll follow this with interest on my travels. If all seem well, feel free
to merge the branch.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Syntax coloring mystery solved!!

['Terry Brown' via leo-editor]

On Sat, 14 Jan 2017 18:22:26 -0800 (PST)
"Edward K. Ream"  wrote:

> Done, and pushed, I think.  It's called fast-colorizing, rev
> 2beba78b5. Let me know if the branch doesn't show up.  This is my
> first branch...

Branch is there in the correct place.

Safe travels.

Cheers -Terry

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Syntax coloring mystery solved!!

[Edward K. Ream]

On Saturday, January 14, 2017 at 9:01:37 PM UTC-5, Edward K. Ream wrote:

Rebecca and I are leaving tomorrow for Madison.  We will be on the road 
> about a week. I would like to up what I have for reference, but I 
> probably can't in the trunk.  Time to create a branch :-)
>

Done, and pushed, I think.  It's called fast-colorizing, rev 2beba78b5. Let 
me know if the branch doesn't show up.  This is my first branch...

It may take quite a bit of tweaking to get all nodes colorized with the 
proper language.  Furthermore, the code needs to be housecleaned. I'll 
attend to all this when I can, but it will likely be next week.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Syntax coloring mystery solved!!

[Edward K. Ream]

On Saturday, January 14, 2017 at 9:52:59 AM UTC-5, Edward K. Ream wrote:
>
> On Sat, Jan 14, 2017 at 7:23 AM, Edward K. Ream  
> wrote:
>
> ​> ​
> Calling QTextDocument.markContentsDirty
> ​ ​
> ​[in the legacy code]​
> does not cause a recoloring.
>
> ​Hurray! Counts show that leo_h.highlightBlock *is* being called the 
> expected number of times. But nothing happens.
>
> A call to viewPort.update() should make the new styles visible.
>

Actually, the jEdit colorizer was not being inited properly.  The colorizer 
is now working well and very quickly.

One bug remains. Switching nodes does not always properly set the 
colorizing language. This is surprisingly difficult, made all the more so 
by subtle differences in which methods get called when.

On my Python 3 machine, it appears that LeoHighlighter.rehighlight is 
*never* called.  That shouldn't be a big problem.

One would think that LeoTree.set_body_text_after_select could properly init 
everything. But bugs still remain.

Rebecca and I are leaving tomorrow for Madison.  We will be on the road 
about a week. I would like to up what I have for reference, but I probably 
can't in the trunk.  Time to create a branch :-)

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Syntax coloring mystery solved!!

[Edward K. Ream]

On Sat, Jan 14, 2017 at 7:23 AM, Edward K. Ream  wrote:

​> ​
Calling QTextDocument.markContentsDirty
​ ​
​[in the legacy code]​
does not cause a recoloring.

​Hurray! Counts show that leo_h.highlightBlock *is* being called the
expected number of times. But nothing happens.

A call to viewPort.update() should make the new styles visible.  I remember
a call something like that in the pyzo code.

It looks like Leo's legacy colorers look can compete on speed with pyzo's.
This is great news. It means that we can use all the legacy mode-related
machinery unchanged.

EKR

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Syntax coloring mystery solved!!

[Edward K. Ream]

​​
​​
On Fri, Jan 13, 2017 at 5:25 PM, 'Terry Brown' via leo-editor <
leo-editor@googlegroups.com> wrote:

​> ​
Congrats on finding the culprit.

​Thank you.

​
​Strangely, the performance bug *remain*s (sometimes) in the legacy
coloring code. I allude to this in this morning's announcement
.
​

​The bug now seems even more mysterious. Calling
QTextDocument.markContentsDirty does not cause a recoloring.  As before,
calling rehighlight directly is super slow.

Note: if a huge node is selected when Leo starts, that node is colored
quickly(!).  But revisiting the node later is slow.

I have spent several hours on this new mystery.  The best clue is that the
body pane's QTextDocument keeps changing with the legacy code, but remains
invariant (as expected) when using pyzo.

Presumably, finding out why the document changes will allow the bug to be
fixed.

​> ​
Couple of things I'd like to see in Leo syntax highlighting, highlighting
of trailing whitespace, and highlighting of https://en.wikipedia.org/
wiki/Non-breaking_space and also of tab (\t).

​The show/toggle-invisibles commands will do some of this.  ​
​Please file a bug report if non-breaking spaces aren't shown to your
satisfaction.

EKR

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Syntax coloring mystery solved!!

['Terry Brown' via leo-editor]

Congrats on finding the culprit.
Couple of things I'd like to see in Leo syntax highlighting, highlighting of 
trailing whitespace, and highlighting of 
https://en.wikipedia.org/wiki/Non-breaking_space and also of tab (\t).  To be 
quite honest I've only made one foray into the syntax highlighting code, some 
time ago, and really can't remember the details.  So don't know if this is hard 
or trivial.
Cheers -Terry
 
  From: Edward K. Ream 
 To: leo-editor  
 Sent: Friday, January 13, 2017 12:23 PM
 Subject: Syntax coloring mystery solved!!
   
tl;dr: Calling doc.markContentsDirty(0, len(p.b)) suffices to recolor a node! 
This should be done instead of calling highlighter.rehighlight explicitly.

I could never have found this bug if I hadn't seen how fast pyzo colorizing 
code was within pyzo itself compared to the same code within Leo. 

Having refresh-from-disk do the coloring correctly and quickly was the crucial 
lucky break.  Tracing showed that a big @edit node was being colorized 
correctly despite highlighter.rehighlight not being called.

After a bit of noodling (without beer) I saw that the body pane's QTextDocument 
must be repeatedly calling highlighter.highlightBlock. I verified this was so 
(without huge traces) by examining the before and after counts of 
highlighter.n_calls.  As hoped, these jumped, indicating that something 
(QTextDocument is the only candidate) was in control.

This is a lucky, lucky break on Friday the thirteenth ;-) It is likely that 
Leo's existing colorizers can be speeded up by 10x. If so, there will be no 
need for the big text hack. Crucially, Qt does not show the text until it has 
been fully colorized.

Furthermore, close study of the pyzo code shows that actual syntax coloring 
could now be deferred until idle time, thereby further improving performance.

Pyzo is full of superb code, and pyzo's colorer is no exception.  I'm not sure 
whether to adapt pyzo's code for all syntax coloring, but I'm tempted.  It 
would be a big project.

In the meantime, I can adapt two good ideas from the pyzo colorer:

1. Precomputing QTextFormat objects saves a lot of time. Leo's present 
colorizers do far too much work withing the main colorizing loop.

2. Using a special tag for unterminated strings is genius.  The three little 
dots are a great visual cue, and terminating the token at the end of the line 
means that adding a quote does not cascade to all following lines.  A great 
hack.

I'm glad this is behind us.  I was starting to doubt my sanity, and starting to 
imagine bugs that would never be found...

The way is now open for more pyzo integration.  I am seriously considering 
replacing Leo's custom text widget with pyzo's.  It a big project, but it will 
pay huge dividends.

Edward
-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.


   
 

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Syntax coloring mystery solved!!

[Edward K. Ream]

*tl;dr:* Calling doc.markContentsDirty(0, len(p.b)) suffices to recolor a 
node! This should be done *instead of* calling highlighter.rehighlight 
explicitly.

I could never have found this bug if I hadn't seen how fast pyzo colorizing 
code was within pyzo itself compared to the *same* code within Leo. 

Having refresh-from-disk do the coloring correctly and quickly was the 
crucial lucky break.  Tracing showed that a big @edit node was being 
colorized correctly despite highlighter.rehighlight *not *being called.

After a bit of noodling (without beer) I saw that the body pane's 
QTextDocument must be repeatedly calling highlighter.highlightBlock. I 
verified this was so (without huge traces) by examining the before and 
after counts of highlighter.n_calls.  As hoped, these jumped, indicating 
that something (QTextDocument is the only candidate) was in control.

This is a lucky, lucky break on Friday the thirteenth ;-) It is likely that 
Leo's existing colorizers can be speeded up by 10x. If so, there will be no 
need for the big text hack. Crucially, Qt does not show the text until it 
has been fully colorized.

Furthermore, close study of the pyzo code shows that actual syntax coloring 
could now be deferred until idle time, thereby further improving 
performance.

Pyzo is full of superb code, and pyzo's colorer is no exception.  I'm not 
sure whether to adapt pyzo's code for all syntax coloring, but I'm 
tempted.  It would be a big project.

In the meantime, I can adapt two good ideas from the pyzo colorer:

1. Precomputing QTextFormat objects saves a lot of time. Leo's present 
colorizers do far too much work withing the main colorizing loop.

2. Using a special tag for unterminated strings is genius.  The three 
little dots are a great visual cue, and terminating the token at the end of 
the line means that adding a quote does not cascade to all following 
lines.  A great hack.

I'm glad this is behind us.  I was starting to doubt my sanity, and 
starting to imagine bugs that would never be found...

The way is now open for more pyzo integration.  I am seriously considering 
replacing Leo's custom text widget with pyzo's.  It a big project, but it 
will pay huge dividends.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.