Hi Alex,
Thanks for responding. It's an interesting idea and I may have to try it.
But I've got 5,000 lines of script that has evolved over 20 years, so I really
don't want to have to rewrite it all! It seems to me that there is an
identifiable problem here; something that is happening on (at least one)
Windows machine, which was fine under LC6 but is broken under LC9; whereas on
Mac it's not the case.
My hope is that there is some simple solution to fix this!
Ben
On 25/08/2021 19:48, Alex Tweedly via use-livecode wrote:
Crazy idea - totally untried .... (sorry, I don't have a Win machine)
put 1 into tLineCount
repeat for each line tRow in tWorkTable
put tRow into tNewTable[tLineCount]
add 1 to tLineCount
end repeat
combine tNewTable using CR
Alex.
On 25/08/2021 18:15, Ben Rubinstein via use-livecode wrote:
Some 20 months ago, I reported that I was in a situation where an app
written in 6.7 needed to be updated to access 64bit drivers, which meant
updating to 9.5 - which displayed horrifying increase in processing time.
In fact I was able to put off the evil day - but now it has returned, and
can be put off no longer. A process that normally takes 2 hours is currently
taking 9. The core processing stage has gone from around ten minutes to over
six hours.
After way too long, I've finally got down to at least one smoking gun; which
is as simple as can be.
Part of what took me so long is a confusion; in production the process runs
on Windows, but I develop on Mac. Although on Mac the overall process does
take about a third longer in LC9 than LC6, the simple tests I've finally
isolated actually run much _quicker_ in LC9 than LC6. So switching between
LC6 and LC9 on Mac as I tried to isolate the issue was giving confusing
signals. But unmistakeably it's *much* slower on Windows.
A simple routine which loops over a load of tab and return formatted data
loaded from a TSV file, to truncate a particular field, had the following
results processing a 70MB file of approximately 257,000 rows:
6.7.11 MacOS 9 seconds
6.7.11 Win32 10 seconds
9.6.3 MacOS 2 seconds
9.6.3 Win32 498 seconds
I simplified it down to this (pointless) loop which just rebuilds a table
one line at a time:
local tNewTable
repeat for each line tRow in tWorkTable
put tRow & return after tNewTable
end repeat
with these results:
6.7.11 MacOS 8 seconds
6.7.11 Win32 7 seconds
9.6.3 MacOS 0 seconds
9.6.3 Win32 591 seconds
(there's obviously a lot of variability in these - both were running in IDE,
on a logged-in computer, so stuff was probably going on in the background;
but I know the overall effect is similar when built as standalone and
running by schedule on an unattended machine. But the key thing is: for this
task, LC9 is dramatically slower on Windows!)
Have others seen something like this?
When I posted about this before (thread: "OMG text processing performance
6.7 - 9.5") Mark Waddingham suggested that it might be to do with a hidden
cost of binary<->text transforms. That makes some sense; but given that the
text already exists, I'm wondering whether taking a line out of text would
cause it to be transformed, only to be transformed again when appending? And
in particular, why this would affect Windows only.
I have also added tests using "is strictly a binary string" in the code
above, and this was true for neither input 'tWorkTable', nor the output
'tNewTable', nor any of the 257,00 extracted lines.
However it is definitely the accumulating of text that is the issue - simply
looping over the lines - even with testing each one to see if it is
"strictly a binary string" - is a second or less on Windows in LC9.
Has anyone had similar experiences? Suggestions for how this could be avoided?
Many thanks in advance,
Ben
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
