Re: OMG text processing performance 6.7 - 9.5
Hi Mark. I have to chime in here that the difference between OS X apps accessing a sql database and Windows doing the same thing in the same app is substantial, and I cannot think why, unless it is Windows itself causing the problem. Querying my customer database for all my customer records takes 14 ticks on OSX. On Windows running in my Parallels it takes 59. On a server 2012 running in VMWare it takes 71! This has made my port to Windows almost unviable. If I had some kind of real answer as to what was causing this kind of lag, it would help. Bob S > On Jan 30, 2020, at 06:04 , Mark Waddingham via use-livecode > wrote: > > > Warmest Regards, > > Mark. > > -- > Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ > LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
RE: OMG text processing performance 6.7 - 9.5
I found this as well. Another thing, it's faster to truncate the string and search from the beginning than using a "start at" on the entire string when searching for all occurrences of a string . This was counter intuitive to me until Mark explained that skipping chars requires more work because repetitive skipping of Unicode chars is slower than many "memcpy"s on very long strings. Ralph DiMola IT Director Evergreen Information Services rdim...@evergreeninfo.net -Original Message- From: use-livecode [mailto:use-livecode-boun...@lists.runrev.com] On Behalf Of Neville via use-livecode Sent: Thursday, January 30, 2020 4:49 PM To: use-livecode@lists.runrev.com Cc: Neville Subject: Re: OMG text processing performance 6.7 - 9.5 Are you perchance using lineOffset searches? I have found that lineOffset performance on utf8 text degrades exponentially with the length of the file, presumably as it searches for line breaks. Use offset instead which remains fast (and much faster still if you can search on the raw text before textencoding, then utf8 encode the found chunks) ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: OMG text processing performance 6.7 - 9.5
Are you perchance using lineOffset searches? I have found that lineOffset performance on utf8 text degrades exponentially with the length of the file, presumably as it searches for line breaks. Use offset instead which remains fast (and much faster still if you can search on the raw text before textencoding, then utf8 encode the found chunks) ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: html5Player
> Richard G. wrote: > do you know if would it be practical to run the generated > LC-engine-as-JS file through a JS->WASM converter? > If not, what would be needed to make that happen? AFAIK the current HTML5 standalone builder was made by Peter Brett in contact with Mark Waddingham. It uses Emscripten (only) to draw to a canvas. So, I don't know but I guess that the HTML5 standalone builder has to be (partially) rewritten to use Emscripten to compile LC source code into WebAssembly modules that run sandboxed together with regular JavaScript (having bindings especially to WebGL). One could omit controls/features that are/will soon be well done in HTML5, for example the audio/video/pdf/camera/img-webP parts. I guess this would run LC in a modern browser for some processes (especially graphics and image processing) much faster than in the current IDE. Who only can really answer (and do) that? Mark Waddingham and his team... ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: html5Player
Hermann, do you know if would it be practical to run the generated LC-engine-as-JS file through a JS->WASM converter? If not, what would be needed to make that happen? -- Richard Gaskin Fourth World Systems Software Design and Development for the Desktop, Mobile, and the Web ambassa...@fourthworld.comhttp://www.FourthWorld.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: html5Player
> Bob S. wrote: Unfortunately I only have the Community version > and cannot test. Works here with LC Community 9.5.1 (on MacOS 10.15.3): Open the stack, select HTML5 (nothing else) in the standalone settings and hit the Test button. You need a HTML5 license *only* for publishing standalones, just the same as you need an Indy license for publishing desktop standalones. [My html5Player is comparable to an Indy standalone that you can download to your OS and open and run local stacks with it. This will run stacks that would run in the IDE.] ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Brave browser
While testing intensively in the LC-HTML5 builder for cross-browser functionality of extensions I used Brave, Chrome, Firefox and Safari. My personal ranking in short: Safari is fastest with huge JS (the standalone engine), Firefox is slowest but has more special features, Chrome has most features (incl. caching technique). But the winner is the open-source browser Brave: Fast as Safari, features as Chrome and runs (close to the same on all OS) on MacOS, Windows, Linux, Android and iOS. https://brave.com/download/ (scroll down for mobile) ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: html5Player
Unnfortunately I only have the Community version and cannot test. Bob S > On Jan 30, 2020, at 10:10 , hh via use-livecode > wrote: > >> Bob S. wrote: >> I probably misunderstand the use of this. >> I drug a simple stack into this but while it displays >> fine, it does not function... > > Did you test it in the IDE? This is very simple in LC 9: > > Open the stack, select HTML5 (nothing else) in the > standalone settings and hit the Test button. > > If it runs there it will run in the LCPlayer because it is > the same engine. > > If it doesn't run then possibly you use wait (not supported > in HTML5, use send in time instead). > Or try to disable your preopenstack handler. > > In case you inject javascript handlers to the loading page > then these may interfere with the main engine. You have to > use a webview with the standalone from a (local) server > instead. > > Known bug: > The only bug I have seen until now with one of my > 100 > tested LC 6/7/8/9 stacks (that don't use unsupported controls > as players) is that a stack without titlebar doesn't 'slip' > into a panel. > One then has to do "close stack " or do "delete stack > " from the LCPlayer-messageBox. > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: html5Player
> Bob S. wrote: > I probably misunderstand the use of this. > I drug a simple stack into this but while it displays > fine, it does not function... Did you test it in the IDE? This is very simple in LC 9: Open the stack, select HTML5 (nothing else) in the standalone settings and hit the Test button. If it runs there it will run in the LCPlayer because it is the same engine. If it doesn't run then possibly you use wait (not supported in HTML5, use send in time instead). Or try to disable your preopenstack handler. In case you inject javascript handlers to the loading page then these may interfere with the main engine. You have to use a webview with the standalone from a (local) server instead. Known bug: The only bug I have seen until now with one of my > 100 tested LC 6/7/8/9 stacks (that don't use unsupported controls as players) is that a stack without titlebar doesn't 'slip' into a panel. One then has to do "close stack " or do "delete stack " from the LCPlayer-messageBox. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: html5Player
Hermann strikes again. Brilliant stuff Hermann - and multiple stacks running at the same time too!! What could you do if you weren't in a straight jacket and blindfolded .(metaphorically speaking). >From memory, you have asked the team for a couple of "tweaks" that would mean you wouldn't have to jump through hoops and workarounds - hope they get some time soon to implement some of your suggestions. Hermann could you list here what is "missing" in a most important to less important list, to make the "webapps" more powerful and easier to "talk" to the DOM, Javascript etc? If you haven't got the time or the inclination I would understand - but maybe just maybe as they are nearing the end of the LCFM project they might look at this sooner rather than later. Regards Lagi On Thu, 30 Jan 2020 at 01:56, hh via use-livecode < use-livecode@lists.runrev.com> wrote: > HTML5: html5Player (v102 as of Jan 30, 2020) > > This is a HTML5 standalone (webApp in the new LC wording). > https://hyperhh.de/html5/html5Player.html > > This is a "standalone-Plus" that is, it is extended by several > javascript extensions for features that are not (yet) implemented > in LC-HTML5 or not possible with LC. > > It can do among other the following. > > * Load (ordinary) local LC stacks by click or drag and drop. > The stacks will run if they would compile and run with the HTML5 > standalone builder (but you don't have to compile). > The stacks MUST have one of the file endings .rev, .livecode or > .livecodescript. > > * Load and display local or remote images (drag and drop images > or copy/paste image urls is supported). > > * Load and display local audio/video (for cross-browser support > use mp3 and mp4 only). Drag and drop file icons is supported. > > * Open a webview for videos, pdfs, audio streams and html pages. > In such a webview you can safely run also HTML5 standalones that > don't work in the html5Player because you inject javascript > handlers/objects to the loading page. > As the main page loads as https you can use https-Addresses only > in the webview (especially for audio/video-streams). > > * Open one or several webcam views (works in newer Chrome, Safari, > Brave, not in Firefox). > > All views are displayed in panels that are draggable and resizable > (incl. minimize and maximize). > > HTML5: html5IDE > (Inspector+Dictionary+ScriptEditor+Tools) > > The above html5Player will complete my experimental html5IDE that > is close to "ready"". I can also meanwhile save edited stacks. But > I don't publish newer versions than > https://hyperhh.de/html5/html5IDE.html > until LC does more in that field than renaming it to "WebApps". > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: html5Player
> BR wrote: can you save anything from that view? 1. Generally yes. But this needs ALWAYS, with all modern browsers, manual user interaction: click a button or select from a menu. Similarly user interaction (click, drag and drop) is needed to read in such objects. * STACKS Panels with an LC icon at top left are stacks: We can save them to the downloads folder (will be implemented in the webIDE), except the Player/IDE itself. Or we can save parts of it (images or text/scripts/prefs) to the downloads folder. * Image/Pdf/Text/Audio/Video Panels with an LC icon at top left are javascript extensions that use callbacks to the main engine: Saving is browser/server dependent. You have the browser interface available for actions with manual user interaction: For example rightClick a PDF to save it to your local filesystem or print it via your OS printing (if the pdf is set to allow this) or download a video/audio (if the audio/video is set to allow this). 2. For the LCPlayer ... ... saving stacks makes sense only for stacks that are downloaded from server and then changed in the LCPlayer, like stack "ImageView" that stores local images for display in a "gallery". But this stack is part of the LCPlayer and will not run correctly without the main stack. Of course you can let the user download stacks that are "modules" (e.g. in a zip) and then import such modules into the LCPlayer (via the "click/drop local stack"). This would work with LC 9-stacks even on every OS that runs the LCPlayer, even on a Raspi 3/4. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: html5Player
Thanks a lot Herman for that work. It opens a lot of possibilities. Kind regards, Alain Vezina > Le 29 janv. 2020 à 20:55, hh via use-livecode > a écrit : > > HTML5: html5Player (v102 as of Jan 30, 2020) > > This is a HTML5 standalone (webApp in the new LC wording). > https://hyperhh.de/html5/html5Player.html > > This is a "standalone-Plus" that is, it is extended by several > javascript extensions for features that are not (yet) implemented > in LC-HTML5 or not possible with LC. > > It can do among other the following. > > * Load (ordinary) local LC stacks by click or drag and drop. > The stacks will run if they would compile and run with the HTML5 > standalone builder (but you don't have to compile). > The stacks MUST have one of the file endings .rev, .livecode or > .livecodescript. > > * Load and display local or remote images (drag and drop images > or copy/paste image urls is supported). > > * Load and display local audio/video (for cross-browser support > use mp3 and mp4 only). Drag and drop file icons is supported. > > * Open a webview for videos, pdfs, audio streams and html pages. > In such a webview you can safely run also HTML5 standalones that > don't work in the html5Player because you inject javascript > handlers/objects to the loading page. > As the main page loads as https you can use https-Addresses only > in the webview (especially for audio/video-streams). > > * Open one or several webcam views (works in newer Chrome, Safari, > Brave, not in Firefox). > > All views are displayed in panels that are draggable and resizable > (incl. minimize and maximize). > > HTML5: html5IDE > (Inspector+Dictionary+ScriptEditor+Tools) > > The above html5Player will complete my experimental html5IDE that > is close to "ready"". I can also meanwhile save edited stacks. But > I don't publish newer versions than > https://hyperhh.de/html5/html5IDE.html > until LC does more in that field than renaming it to "WebApps". > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: OMG text processing performance 6.7 - 9.5
Ben, what DB are you connecting to? We are running here a VM with Windows 2019 and MS SQL 2017. On a Windows 10 64bit VM we are using the 32 bit Microsoft ODBC Driver 11 for SQL Server to connect from our 32bit LC standalone to the MSSQL server, although 64bit ODBC Driver 11 is installed. But i cannot remember, if the 32bit driver was installed separately or was automatically installed when the 64bit ODBC Driver11 was installed. - Matthias Rebbe Life Is Too Short For Boring Code > Am 30.01.2020 um 14:20 schrieb Ben Rubinstein via use-livecode > : > > I'm looking for a hints about where the speed has gone in the current > (Unicode era) LiveCode text processing. I've been vaguely aware that text > processing performance suffered in the transition, but haven't needed to > focus on it before. > > The context is that I'm finally forced to replace an app that's been > processing data for a client for well over a decade. To date the standalone > has been built on LC 6.7.11; but now we need to put it on a new platform with > 64-bit database drivers. The performance has gone through the floor, through > the floors below, through the foundations, and is on its way to the centre of > the earth. > > The first stage of the app - which retrieves a load of data from various > databases and online sources, does minimal processing on it, and dumps it to > cache files - is approx 2x slower. The main core of the app, which loads this > data in and does a vast amount of processing on it to generate various output > data and reports, has gone from 12 minutes to over *six hours*. > > (The server itself is different, and running Windows Server 2016 rather than > Windows Server 2008, rather than but they're both VMs, quite likely on the > same underlying hardware, and if anything I'd expect the new server to be > more performant. Of course I assume that a new version of the OS will always > be slower ) > > The coding is gnarly - the oldest parts are probably at least 15 years old - > and I've no doubt it could be made more efficient; but we don't have time or > budget to rewrite it all. So, are there known gotchas, functions which have > taken a much greater hit than others, that I could concentrate on to get the > most ROI in speeding this up? > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode - Matthias Rebbe Life Is Too Short For Boring Code ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: html5Player
Thanks Hermann. I probably misunderstand the use of this. I drug a simple stack into this but while it displays fine, it does not function. It is a simple Time Calculator where I enter the number of devices, a start and stop time with a lunch break and it divides the time "on the clock" between each device to the nearest minute. The Reset button which puts empty in all the data in the fields doesn't even do that. Bob S > On Jan 29, 2020, at 17:55 , hh via use-livecode > wrote: > > HTML5: html5Player (v102 as of Jan 30, 2020) > > This is a HTML5 standalone (webApp in the new LC wording). > https://hyperhh.de/html5/html5Player.html > > This is a "standalone-Plus" that is, it is extended by several > javascript extensions for features that are not (yet) implemented > in LC-HTML5 or not possible with LC. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: OMG text processing performance 6.7 - 9.5
On 2020-01-30 14:38, Ben Rubinstein via use-livecode wrote: Hi Mark, Thanks for taking the time to reply! I'm indeed currently in the process of seeing whether I can persuade the client's IT department to install the 32-bit drivers on the new VM. I'm optimistic that will buy me some time, but it won't be a complete solution because they outsource support to a third company, which has warned that it doesn't intend to support the 32-bit drivers much longer (apparently they're just waiting for Crystal Reports to be updated!). Ah! From that I'm guessing you are using the ODBC revdb driver - which needs a third-party ODBC connector :) And if that fails, one of my options is as you suggest to use the LC 9.5-built app to retrieve the data through the 64-bit drivers, and the the LC 6.7-built app to process and (probably) build it. It will be shonky. It doesn't have to be 'shonky' - if the fetch-from-database part is already separated from the data-processing-part through cache-files (i.e. fetch writes to files on disk, data-process reads said files and processes) then you could build a 64-bit win standalone which is the fetch-from-database part, which is then called by the data-process part using shell (or open process). Of course, it would be slightly cleaner to all be one app :) However, what you say certainly makes me feel more optimistic that something should be possible. There's really very little going on in the way of binary<->text conversion; there probably is a fair amount of word chunking - although half the work is about tracing cross-references etc, there's also a fair amount of processing of 'prose' and prose-like text. However, the nature of the text is that although 99% of it is probably ASCII, in any given table of text there will be just a few 'extended' characters - does that mean it all gets treated as four-byte data? Binary<->text can be quite subtle - as it isn't something you had to think about before 6.7. For example, if you are fetching using *b via revDB from the database, then *that* will now be binary data - not text. (Indeed, what accessors are you using to get the data?) Also, things like binfile and reading for binary (from a file) will also produce binary rather than text. You can test for binary data using 'is strictly a binary string'. Native encoding means (on Windows at least) anything which fits into Latin-1 so any text you are getting out of revDB from the database should come through as native strings. Native strings get converted to unicode internally when they are combined with a string which contains unicode and in two other places: 1) Using matchText / replaceText (because we use the utf-16 variant of PCRE) 2) When put into a field (because all text layout APIs on all platforms use UTF-16) What sort of text operations are you using for 'tracing cross-references etc' and 'processing of 'prose' and prose-like text'? I'll see how the negotiations with IT get on... Good luck! Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: OMG text processing performance 6.7 - 9.5
Hi Mark, Thanks for taking the time to reply! I'm indeed currently in the process of seeing whether I can persuade the client's IT department to install the 32-bit drivers on the new VM. I'm optimistic that will buy me some time, but it won't be a complete solution because they outsource support to a third company, which has warned that it doesn't intend to support the 32-bit drivers much longer (apparently they're just waiting for Crystal Reports to be updated!). And if that fails, one of my options is as you suggest to use the LC 9.5-built app to retrieve the data through the 64-bit drivers, and the the LC 6.7-built app to process and (probably) build it. It will be shonky. However, what you say certainly makes me feel more optimistic that something should be possible. There's really very little going on in the way of binary<->text conversion; there probably is a fair amount of word chunking - although half the work is about tracing cross-references etc, there's also a fair amount of processing of 'prose' and prose-like text. However, the nature of the text is that although 99% of it is probably ASCII, in any given table of text there will be just a few 'extended' characters - does that mean it all gets treated as four-byte data? I'll see how the negotiations with IT get on... Ben On 30/01/2020 14:04, Mark Waddingham via use-livecode wrote: On 2020-01-30 13:20, Ben Rubinstein via use-livecode wrote: The context is that I'm finally forced to replace an app that's been processing data for a client for well over a decade. To date the standalone has been built on LC 6.7.11; but now we need to put it on a new platform with 64-bit database drivers. The performance has gone through the floor, through the floors below, through the foundations, and is on its way to the centre of the earth. What's the need for 64-bit database drivers? i.e. What are you currently using to talk to the database and why can you not continue to use a 32-bit Windows standalone? The first stage of the app - which retrieves a load of data from various databases and online sources, does minimal processing on it, and dumps it to cache files - is approx 2x slower. The main core of the app, which loads this data in and does a vast amount of processing on it to generate various output data and reports, has gone from 12 minutes to over *six hours*. I suspect it is probably a couple of things which are being done uniformly causing the problem rather than lots of things all over the place... Where exactly is the data coming from? (at a high-level) what sorts of operations are being performed on it? what sort of I/O is being performed? The main one I can think of is implicit binary<->text conversions. In 6.7 and below binary data and text were the same thing - in 7+ they are distinct types which require a conversion operation. The functions which were always really returning/taking binary data now actually do. e.g. textEncode / Decode, compress / decompress, binaryEncode / binaryDecode, the byte chunk, repeat for each byte, numToByte Given the app is coming from 6.7 vintage, it is unlikely that any of the new unicode text codepaths would be hit (unless there's something odd going on somewhere) as binary data converts to native encoded text - unless of course the means by which the data is getting into the app is being taken as unicode strings (without knowing the exact I/O going on I can't really see how this could happen, but I can't rule it out). In general, native text processing (item detection, comparison, containment and such) is all as fast if not faster in the post-7 engines than 6.7 as I spent quite a while specializing a lot of lower level routines to make sure it was. I do know the word chunk has been somewhat adversely affected, however, as that was never optimized in the same way. The coding is gnarly - the oldest parts are probably at least 15 years old - and I've no doubt it could be made more efficient; but we don't have time or budget to rewrite it all. So, are there known gotchas, functions which have taken a much greater hit than others, that I could concentrate on to get the most ROI in speeding this up? Given that you don't have time nor budget to really touch the code at all in any depth then it would best to not have to touch it at all and keep it in 6.7.11? i.e. Do you really need to move to 6? Could you split the app into the bit which does the database communication and caching (assuming that *really* needs to be 64-bit) and the bit which does the data processing (which could remain as 32-bit in 6.7.11). Note I should say that the reason I ask the above is not because of a lack of confidence in getting your code to run as fast as it did before but because of pure business reasoning - why spend time and money on something which isn't necessarily really needed? There's a difference between needing to update user-facing apps and true back-office server apps after all - banks and
Re: OMG text processing performance 6.7 - 9.5
On 2020-01-30 13:20, Ben Rubinstein via use-livecode wrote: The context is that I'm finally forced to replace an app that's been processing data for a client for well over a decade. To date the standalone has been built on LC 6.7.11; but now we need to put it on a new platform with 64-bit database drivers. The performance has gone through the floor, through the floors below, through the foundations, and is on its way to the centre of the earth. What's the need for 64-bit database drivers? i.e. What are you currently using to talk to the database and why can you not continue to use a 32-bit Windows standalone? The first stage of the app - which retrieves a load of data from various databases and online sources, does minimal processing on it, and dumps it to cache files - is approx 2x slower. The main core of the app, which loads this data in and does a vast amount of processing on it to generate various output data and reports, has gone from 12 minutes to over *six hours*. I suspect it is probably a couple of things which are being done uniformly causing the problem rather than lots of things all over the place... Where exactly is the data coming from? (at a high-level) what sorts of operations are being performed on it? what sort of I/O is being performed? The main one I can think of is implicit binary<->text conversions. In 6.7 and below binary data and text were the same thing - in 7+ they are distinct types which require a conversion operation. The functions which were always really returning/taking binary data now actually do. e.g. textEncode / Decode, compress / decompress, binaryEncode / binaryDecode, the byte chunk, repeat for each byte, numToByte Given the app is coming from 6.7 vintage, it is unlikely that any of the new unicode text codepaths would be hit (unless there's something odd going on somewhere) as binary data converts to native encoded text - unless of course the means by which the data is getting into the app is being taken as unicode strings (without knowing the exact I/O going on I can't really see how this could happen, but I can't rule it out). In general, native text processing (item detection, comparison, containment and such) is all as fast if not faster in the post-7 engines than 6.7 as I spent quite a while specializing a lot of lower level routines to make sure it was. I do know the word chunk has been somewhat adversely affected, however, as that was never optimized in the same way. The coding is gnarly - the oldest parts are probably at least 15 years old - and I've no doubt it could be made more efficient; but we don't have time or budget to rewrite it all. So, are there known gotchas, functions which have taken a much greater hit than others, that I could concentrate on to get the most ROI in speeding this up? Given that you don't have time nor budget to really touch the code at all in any depth then it would best to not have to touch it at all and keep it in 6.7.11? i.e. Do you really need to move to 6? Could you split the app into the bit which does the database communication and caching (assuming that *really* needs to be 64-bit) and the bit which does the data processing (which could remain as 32-bit in 6.7.11). Note I should say that the reason I ask the above is not because of a lack of confidence in getting your code to run as fast as it did before but because of pure business reasoning - why spend time and money on something which isn't necessarily really needed? There's a difference between needing to update user-facing apps and true back-office server apps after all - banks and insurance companies still have software written on and running on machines which are decades old because they work and the cost of keeping them running is vastly less than the cost to rewrite and replace!). Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: html5Player
Herman: can you save anything from that view? I always have to remind myself that it is Livecode running in a browser. It feels like some dedicated app. I suggest that anybody who is interested in what is possible with Livecode HTML5 to have a look at Hermann's examples. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
OMG text processing performance 6.7 - 9.5
I'm looking for a hints about where the speed has gone in the current (Unicode era) LiveCode text processing. I've been vaguely aware that text processing performance suffered in the transition, but haven't needed to focus on it before. The context is that I'm finally forced to replace an app that's been processing data for a client for well over a decade. To date the standalone has been built on LC 6.7.11; but now we need to put it on a new platform with 64-bit database drivers. The performance has gone through the floor, through the floors below, through the foundations, and is on its way to the centre of the earth. The first stage of the app - which retrieves a load of data from various databases and online sources, does minimal processing on it, and dumps it to cache files - is approx 2x slower. The main core of the app, which loads this data in and does a vast amount of processing on it to generate various output data and reports, has gone from 12 minutes to over *six hours*. (The server itself is different, and running Windows Server 2016 rather than Windows Server 2008, rather than but they're both VMs, quite likely on the same underlying hardware, and if anything I'd expect the new server to be more performant. Of course I assume that a new version of the OS will always be slower ) The coding is gnarly - the oldest parts are probably at least 15 years old - and I've no doubt it could be made more efficient; but we don't have time or budget to rewrite it all. So, are there known gotchas, functions which have taken a much greater hit than others, that I could concentrate on to get the most ROI in speeding this up? ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: html5Player
Hermann, This is magic. I always have to remind myself that it is Livecode running in a browser. It feels like some dedicated app. I suggest that anybody who is interested in what is possible with Livecode HTML5 to have a look at Hermann's examples. https://hyperhh.de/html5/html5Player.html and https://hyperhh.de/html5/html5IDE.html Take a little time to explore the multitude of features. Thank you for making this available Kind regards Bernd ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
