Hi Mark,
Thank you for this, very promising.
Only two things puzzled me. One you've already addressed when you corrected
the specified cheese.
The other was
The only caveat is that it might cause apps mutating lots of medium->large
strings concurrently to take up more memory in general... > ... (and any issue arising from that could be resolved by moving to the 64-bit
windows engine).
I have been doing my tests on the 64-bit Windows engine. What am I missing?
TIA (and T as always for all your work),
Ben
On 10/09/2021 14:06, Mark Waddingham via use-livecode wrote:
On 2021-09-02 18:38, Mark Waddingham via use-livecode wrote:
We will endeavour to fix for 9.6.5-rc-1 (due 'real soon now'!).
So I have been prodding the windows 'accumulating large strings' speed problem
this week (in amongst other things).
It is definitely memory allocation causing the problem.
The rule for extending a string buffer when inserting/appending more text is
pretty much the same as 6.7 (the amount of 'extra space' it asks for each time
is a little less - but changing it to the constant used in 6.7 makes no
difference). What is different between 6.7 and 7.0+ is the memory allocation
patterns of the engine itself and I think it is this which is invariably
causing a whole new buffer having to be allocated when appending to large
strings, rather than the existing one being extended.
The windows heap is much more prudent than UNIXy counterparts it would seem -
where UNIX heaps will happily leave plenty of free space (which the heaps know
about and thus can re-use), Windows appears to avoid that like the plague
(which I'm sure is the case for lots of historical reasons and backwards
compatibility). [ To give a very rough analogy, the map of used space in a
heap on windows is like a block of cheddar; whereas on UNIXy systems it will
be like a block of edam ].
So anyway, long story short, making a string buffer grow in proportion to its
size appears to mitigate the problem - I still need to finesse things a bit
though to ensure that memory starvation doesn't occur when you have a couple
of large mutating strings but all being well we should be able to get
something together for 9.6.5-rc-1.
The only caveat is that it might cause apps mutating lots of medium->large
strings concurrently to take up more memory in general (i.e. in that extra
unused 'just in case' space at the end of each buffer) but no more than the
same apps would on any other (UNIX-based) platform (and any issue arising from
that could be resolved by moving to the 64-bit windows engine).
For reference, the bug about the string accumulation problem as posed by Ben
is https://quality.livecode.com/show_bug.cgi?id=23319 (this will also fix the
sort speed too - as the final step the sort command performs internally is
re-accumulating the string in the new order).
Warmest Regards,
Mark.
P.S. I cannot really say whether this will help with the various 'IDE can be
like treacle' on Windows problems - but it definitely can't hurt!
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
