[tw5] Re: [tw] Re: The Performance At Scale Bummer ... Initial Notes

[@TiddlyTweeter]

The discussion in this thread is interesting to me.

At a practical level its, perhaps, worth noting that in the last year I put 
three projects "on hold" because TW testing indicated it could not cope.

FWIW, they are ...

1 - 50,000 simple sections from the Perry Mason TV series scripts. The idea 
here is to display 100 tiddlers of quotes at a time. And let readers 
favourite what grabs their interest.

2 - A complex project for Dickens' novel Great Expectations that is chunked 
down to paragraphs, with annotations at paragraph level, cross linking to 
another wiki for clips from thirteen versions of GE that have been filmed, 
and a Glossary of terms used.

3 - Chaucer's Canterbury Tales in middle English with line by line 
commentary.

All three TW can do what is needed very well, easily. Except that NONE of 
them can cope with the SCALE. In practice I can't get it responsive enough 
to make any of them public.

Just saying
Josiah

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/958d1e6d-ceba-4f88-85c9-94d5ab896b37%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: The Performance At Scale Bummer ... Initial Notes

[BurningTreeC]

>
> On Saturday, April 7, 2018 at 9:21:49 AM UTC+2, BurningTreeC wrote:
>>
>> From the second link, requestIdleCallback() appears to be very 
>> interesting - if the browser support is given. It reschedules workloads to 
>> idle-time like requestAnimationFrame()
>>
>
> https://caniuse.com/#search=requestIdleCallback
>
> seems to be a no-go for me. 
>
> -m
>

also when only using it when available, so that someone that wants to do 
workload-heavy things could use one of the compatible browsers (latest 
chrome/chromium/tiddlydesktop/firefox/brave/beakerbrowser ... )?

Simon

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/654b98a1-ddd6-4070-a5fd-7b09ccd1ac0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tw] Re: The Performance At Scale Bummer ... Initial Notes

[PMario]

On Saturday, April 7, 2018 at 11:48:55 AM UTC+2, Jed Carty wrote:
>
> 
>
 

> I have been considering trying to use TaffyDB (http://taffydb.com 
> )
>  
> with tiddlywiki but I haven't had a chance to try it out yet and I have no 
> idea if it would actually help at all. Another option is to use an 
> in-browser sqlite database and maybe web workers.
>

Have you had a look at the code. ... For me it seems to be a wrapper for 
jsavascript object handling. ... I doubt it would make anything fast. 

TiddlyWiki's utils.js does the same things and is compatible with TW. 

-m

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/ef097164-2a70-42b0-9421-2f106ed82d8c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: The Performance At Scale Bummer ... Initial Notes

[PMario]

On Saturday, April 7, 2018 at 9:21:49 AM UTC+2, BurningTreeC wrote:
>
> From the second link, requestIdleCallback() appears to be very interesting 
> - if the browser support is given. It reschedules workloads to idle-time 
> like requestAnimationFrame()
>

https://caniuse.com/#search=requestIdleCallback

seems to be a no-go for me. 

-m

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/6f1db257-7472-470e-9dec-b71412968b3a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tw] Re: The Performance At Scale Bummer ... Initial Notes

[Jed Carty]

Looking through my things, despite using the Object.keys().forEach 
structure many times there is only one time I have found that may lead to 
any sort of performance improvement. Most of the uses are over a very small 
list (only one or two items generally) and only happen rarely (like when a 
tiddler is saved). It is the same with using delete. It is used to remove 
entries about tiddlers when they are deleted, which happens relatively 
rarely and is in a context where the alternative (setting the value to 
undefined instead of deleting it) isn't a viable solution because it would 
cause more noticeable problems elsewhere. Checking to see if a property is 
listed is quicker than checking to make sure that the property is listed 
with a value other than undefined, and checking for the existence of a 
tiddler happens much more often than deleting a tiddler.

I always get excited when I see things like this that sound like they can 
give a big performance boost but almost every time it turns out to be 
optimisations that would just be lost in the other processing. Even if 
there is a 99% performance increase in the parts that are affected by these 
(the object.keys loops and delete) I don't think that the overall 
performance of the wiki would be noticeably changed, neither one is 
directly related to the changes in performance as the number of tiddlers in 
a wiki increases.

I think to fix the problem of scalability we probably need to look at a 
plugin that moves the tiddler store into an in-browser database and do 
filtering based on queries to that instead of having everything held in 
memory as a json object. But there is almost certainly going to be a 
trade-off there, for normal use where a big wiki has a few hundred tiddlers 
than adding on a database engine would probably make the wiki slower. I 
have been considering trying to use TaffyDB (http://taffydb.com) with 
tiddlywiki but I haven't had a chance to try it out yet and I have no idea 
if it would actually help at all. Another option is to use an in-browser 
sqlite database and maybe web workers.

I suspect that using a database this way may require a re-write of all the 
existing filter operators. I am not sure, this is all speculation.

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/78537423-7b18-4d7b-b42b-fe2cb753ded9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: The Performance At Scale Bummer ... Initial Notes

[BurningTreeC]

In the first link I posted it shows something notable about using "delete" 
for deleting object properties: 

In V8 6.0 and 6.1 (not yet used in any Node releases), deleting the last 
property added to an object hits a fast path in V8, and thus it is faster, 
even, than setting to undefined. This is excellent news, as it shows that 
the V8 team is working towards improving the performance of delete. 
However, the delete operator will still result in a significant performance 
hit on property access if a property *other than the most recently added 
property* is deleted from the object. So overall we have to continue to 
recommend against using delete.

I've seen that it's used within the tiddlywiki core and the mentioned 
discovery appears to be a recent one


@Jed Carty

It's a bit annoying not knowing exactly where it goes and which tweaks made 
now are future-proof
I hope that we can filter those tweaks that make sense so that we won't 
need to change again in half a year

Simon

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/7751b600-26a5-4348-bbd9-7516fb85fdf9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tw] Re: The Performance At Scale Bummer ... Initial Notes

[Jed Carty]

Oh, that is unfortunate. I use Object.keys.forEach a lot in my plugins 
because it is convenient for me to type. Also reading about how the 
performance of for (property in object) {..} type loops is changing is 
interesting. It is going to be less efficient than the 
object.keys().foreach option in node 8 due to some serious performance hits 
to it and then in the next version it should go back to being much faster 
(in node, in the browser from what I can see it is just faster).

It isn't a difficult change so I guess I am going to be spending some time 
today updating plugins.

Another thing that I have been looking at is not sending plugins to the 
browser if they only affect the node process. For the core tiddlywiki 
plugins I think that this is just the filesystem plugin but for what I have 
been working on there have been more. This probably won't make a difference 
in the normal performance of a wiki but it should make the file loaded in 
the browser smaller and hopefully give a shorter load time.

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/547f3ed1-aa54-44bc-9474-d648f329537c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: The Performance At Scale Bummer ... Initial Notes

[BurningTreeC]

I found two interesting pages that are quite up-to-date and talk about 
javascript performance on nodejs (v8) and the second on firefox:


   - 
   https://www.nearform.com/blog/node-js-is-getting-a-new-v8-with-turbofan/
   - 
   
https://developer.mozilla.org/en-US/Firefox/Performance_best_practices_for_Firefox_fe_engineers
   
>From the second link, requestIdleCallback() appears to be very interesting 
- if the browser support is given. It reschedules workloads to idle-time 
like requestAnimationFrame()
This page shows a shim if there's no support for 
it: https://developers.google.com/web/updates/2015/08/using-requestidlecallback

I tested my latest chrome, firefox, brave browser and beaker browser and 
they're all supporting it
I just put this in the developer console: 

if ('requestIdleCallback' in window) {
  console.log("yes");
} else {
  console.log("no");
}


This benchmark https://jsperf.com/object-keys-vs-for-in-with-closure/10  
tells me that Object.keys with a for loop - and for (key in object) are 
pretty much the fastest cross-browser
It also suggests that Object.keys().forEach ... should be avoided, it's 44% 
slower than the fastest in my benchmark


Simon

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/7b9ef087-7708-42a0-b0fa-c820f76aff39%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: The Performance At Scale Bummer ... Initial Notes

[Jeremy Ruston]

Hi Mark

> My first attempt to import the tiddlers into a pre-release version crashed 
> the browser tab. So I thought that maybe I needed to use node to render out 
> the tiddlers as tids. Now we hit another scalability wall. The process has 
> been running 30 minutes and isn't even a quarter of the way through. Looks 
> like converting to tids will take two hours!

The approach that worked on my ancient laptop was to open James’ original 
TiddlySpot wiki in Chrome and then export a JSON file of all tiddlers. Then on 
the command line I built the new wiki with:

tiddlywiki editions/empty/ --load ~/Downloads/wardjhtrumptweets.json --build 
index

However, I found that performance was drastically *worse* in 5.1.16, which I am 
now investigating...

> Next question, is the entire master branch on github considered the 
> pre-release?

Yes, the prerelease is just the Git master.

Best wishes

Jeremy.

> 
> -- Mark
> 
> On Friday, April 6, 2018 at 7:37:19 AM UTC-7, @TiddlyTweeter wrote:
> BTC
> 
> Good thought. I'll point the author to your suggestion.
> 
> TT
> 
> BurningTreeC wrote:
> For a good comparison I would like to see that big one using the latest 
> prerelease version, where some performance tweaks have been done
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to tiddlywiki+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to tiddlywiki@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/tiddlywiki 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/tiddlywiki/5e037fd3-3d96-49c8-b006-8072e34f7382%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/139225A2-9B63-4319-B1E7-C2E370236693%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[tw] Re: The Performance At Scale Bummer ... Initial Notes

['Mark S.' via TiddlyWiki]

My first attempt to import the tiddlers into a pre-release version crashed 
the browser tab. So I thought that maybe I needed to use node to render out 
the tiddlers as tids. Now we hit another scalability wall. The process has 
been running 30 minutes and isn't even a quarter of the way through. Looks 
like converting to tids will take two hours!

Next question, is the entire master branch on github considered the 
pre-release?

-- Mark

On Friday, April 6, 2018 at 7:37:19 AM UTC-7, @TiddlyTweeter wrote:
>
> BTC
>
> Good thought. I'll point the author to your suggestion.
>
> TT
>
> BurningTreeC wrote:
>>
>> For a good comparison I would like to see that big one using the latest 
>> prerelease version, where some performance tweaks have been done
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/5e037fd3-3d96-49c8-b006-8072e34f7382%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tw] Re: The Performance At Scale Bummer ... Initial Notes

[wardjh]

I will have the larger project ported over to the newest version of TW this 
weekend and run it off my personal server. 
For the smaller project I am  going to keep adding 100 more records at a 
time in the spreadsheet until I notice that it starts to slow down. 

Thanks, 



On Friday, April 6, 2018 at 10:37:19 AM UTC-4, @TiddlyTweeter wrote:
BTC

>
> Good thought. I'll point the author to your suggestion.
>
> TT
>
> BurningTreeC wrote:
>>
>> For a good comparison I would like to see that big one using the latest 
>> prerelease version, where some performance tweaks have been done
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/a4d1e4e5-0185-490a-8cf2-75308b0b4616%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tw] Re: The Performance At Scale Bummer ... Initial Notes

[@TiddlyTweeter]

BTC

Good thought. I'll point the author to your suggestion.

TT

BurningTreeC wrote:
>
> For a good comparison I would like to see that big one using the latest 
> prerelease version, where some performance tweaks have been done
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/2137f815-1ac1-4752-a903-cd36a2386552%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tw] Re: The Performance At Scale Bummer ... Initial Notes

[BurningTreeC]

Hi @TiddlyTweeter 

For a good comparison I would like to see that big one using the latest 
prerelease version, where some performance tweaks have been done

BTC

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/6b8394fc-ccfd-41d2-9f81-5fca4c31cbd4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.