Re: custom property searching speed question
Ben Fisher wrote: What algorithm is RR using for searching? I find that I have to keep writing Boyer-Moore-Sunday search algorithms for most of the devlopment tools I use. Whatever algorithm RR uses to search the docs, it doesn't work. When I try to Search for: something, I get a spinning beach ball for a while, and then Rev stops responding. There are a couple of searching problem in the docs - Ken's FixDocs is a temp patch for one of them, and there's another one in the Find Replace function; Chipp's fix handles that one Fix for Rev 2.5 FindReplace stack. Turns out there's a problem with shared group references. It manifests itself in a spinning busy icon which never stops and no hits seen in the fields below. I've created a patch stack: go URL http://www.altuit.com/webs/altuit2/RunRev/altRevSearchFix.rev; But I think the original question was about the search algorithm used for offset() and similar functions. I don't know whether these use a Boyer-Moore or not. I do know they are very fast. Given the relative speed of scripted code versus library code, it would be astonishing if it was possible to script in Transcript a B-M solution that would approach the speed of the built-in functions. Mikey (I think it was) - have you encountered any speed problems that you think a faster search would be needed to solve ? -- Alex Tweedly http://www.tweedly.net No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.308 / Virus Database: 266.6.0 - Release Date: 02/03/2005 ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
Re: custom property searching speed question
What algorithm is RR using for searching? I find that I have to keep writing Boyer-Moore-Sunday search algorithms for most of the devlopment tools I use. -- On the first day, God created the heavens and the Earth On the second day, God created the oceans. On the third day, God put the animals on hold for a few hours, and did a little diving. And God said, This is good. ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
Re: custom property searching speed question
What algorithm is RR using for searching? I find that I have to keep writing Boyer-Moore-Sunday search algorithms for most of the devlopment tools I use. Whatever algorithm RR uses to search the docs, it doesn't work. When I try to Search for: something, I get a spinning beach ball for a while, and then Rev stops responding. Also it is funny how the docs mention the Animation Builder several times, without ever once saying it is gone. Seriously considering a bugzilla, Ben ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
Re: custom property searching speed question
On 3/3/05 10:03 PM, Ben Fisher wrote: Whatever algorithm RR uses to search the docs, it doesn't work. When I try to Search for: something, I get a spinning beach ball for a while, and then Rev stops responding. This has been corrected for the next release. Ken Ray posted a temporary fix for it though, which you can find in his user space on RevOnline at kray(FixDocs). -- Jacqueline Landman Gay | [EMAIL PROTECTED] HyperActive Software | http://www.hyperactivesw.com ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
custom property searching speed question
Say you had a custom property set - theProps And the keys of this set were 1 through 500,000 So, basically, you had an array with half a million elements stored as a custom property set. Then, you wanted to search that array, and do it in such a way that the search returned the name and content of the first custom property that contains the item for which you search. Would the following method be fastest? Set the custompropertyset to theProps Put 0 into Z Repeat for each element E in the customproperties of myobject Add 1 to Z If E contains searchterm then exit repeat End repeat Put Z the customproperties[Z] of myobject into field feedback My questions are this: 1) Is there a better or faster-access way to store the array than as a custom property set? 2) Is there a search method that is faster than doing all those comparisons in transcript? For example, lineoffset and itemoffset are supposed to be very fast. Is it possible to use itemoffset on an array, or is there anything that works like an elementoffset command would work, if it existed? 3) Would it be faster to combine the array, and use itemoffset? 4) Could filter be made to work in this situation, or would it only give the value of the element, but not the name of the element? 5) Anything faster that I am not thinking of? Thanks, Jonathan -- ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
Re: custom property searching speed question
Lynch, Jonathan wrote: Say you had a custom property set - theProps And the keys of this set were 1 through 500,000 So, basically, you had an array with half a million elements stored as a custom property set. Then, you wanted to search that array, and do it in such a way that the search returned the name and content of the first custom property that contains the item for which you search. Would the following method be fastest? Set the custompropertyset to theProps Put 0 into Z Repeat for each element E in the customproperties of myobject Add 1 to Z If E contains searchterm then exit repeat End repeat Put Z the customproperties[Z] of myobject into field feedback My questions are this: 1) Is there a better or faster-access way to store the array than as a custom property set? Using repeat for each line on a string benchmarked about 20% faster here than using repeat for each element on an array. I only tested on 40,000 lines, though. In general repeat for each... scales well, so I would feel fairly confident extrapolating my results to larger data sets. 2) Is there a search method that is faster than doing all those comparisons in transcript? For example, lineoffset and itemoffset are supposed to be very fast. Is it possible to use itemoffset on an array, or is there anything that works like an elementoffset command would work, if it existed? 3) Would it be faster to combine the array, and use itemoffset? Offset can rip through large blocks of text very fast, but it's not very precise. For example, in my case I had to do comparisons on specific items within a line. LineOffset will get you to that line, but won't tell you where within that line it is. With a low number of hits lineOffset can be faster to find the line, and then you could evaluate specific elements to find the item if needed. But for larger numbers of hits it should be slower, since once you find the line you still need to get line x, and that requires the engine to count lines. Requiring the engine to count lines is the bottleneck. So for my purposes, using repeat for each line gave me a consistently scalable solution which allowed me to query any items within a line without ever having to count lines. So lazy person that I am, I stopped there and moved on to other things. :) 4) Could filter be made to work in this situation, or would it only give the value of the element, but not the name of the element? Filter may benchmark the fastest if you're looking for an item anywhere in a line (never tried it myself, since I need to find matches for specific items within lines, but worth testing). 5) Anything faster that I am not thinking of? Probably. Search algorithms are a deep subject, and there's always one more clever way to solve a given problem. I stopped benchmarking these things once I found that repeat for each line was coming out okay. Because I have so many comparisons to perform on specific items within a line, it gave me a robust (though admittedly brute force) solutuion with acceptable performance on data sets larger than will be needed in real-world performance with my app's audience. But with half a million records it begs the question: Why not consider a database, where searching is done by compiled code optimized by people who specialize in such things? -- Richard Gaskin Fourth World Media Corporation __ Rev tools and more: http://www.fourthworld.com/rev ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
RE: custom property searching speed question
Does filter work on an array? Anyway, I experimented with a customproperty set created to contain 500,000 elements - each element contains like 7 words. Only the very last element contains the word nonstandard So far, the following script is fastest: on mouseUp put the milliseconds into M put field search text into ST put the customproperties of field theData into myArray repeat for each element E in myArray if matchchunk(E,ST) = true then put E into field output exit repeat end if end repeat put the milliseconds into M2 put (M2 - M)/1000 into field feedback end mouseUp Each element was created so that the name of that element is the first word of the element - meaning I do not have to keep track of which element we are dealing with - as long as I have the value of the element, I automatically have the name of the element as well. If the word being searched for is in the 500,000th element of the custom property set, then it takes 1.059 seconds to find (with like 4 other programs running on my computer at the same time) If the elements are already combined into a single variable, then lineoffset takes just as long! I guess repeat for each is exceedingly fast. If I could figure out how to use filter on an array - and how to get it to work in the same way as using contains then I would test it with the filter command - but so far this eludes me. In my experiements with this, it seems that an if-then structure was faster than a switch structure - is this always true? ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
RE: custom property searching speed question
More tinkering and testing of timing... I changed the customproperty set so that it contains 500,000 records with 3 items in each record. Item 1, the name of the element Item 2, a bunch of words Item 3, a number (100) I created a script that gets the sum of column 3 of this data set - that is, it adds up item 3 for each element. This script: on mouseUp put the milliseconds into M put the customproperties of field theData into myArray put 0 into tSum set the itemdelimiter to numtochar(30) repeat for each element E in myArray add item 3 of E to tSum end repeat put tSum into field output put ((the milliseconds)-M)/1000 into field feedback end mouseUp produces the correct result in .486 seconds! Half a second to sum a column of 500,000 items! For comparison purposes - how fast would summing up the third column for half-a-million records get done in a professional database program? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Lynch, Jonathan Sent: Wednesday, March 02, 2005 5:03 PM To: How to use Revolution Subject: RE: custom property searching speed question Does filter work on an array? Anyway, I experimented with a customproperty set created to contain 500,000 elements - each element contains like 7 words. Only the very last element contains the word nonstandard So far, the following script is fastest: on mouseUp put the milliseconds into M put field search text into ST put the customproperties of field theData into myArray repeat for each element E in myArray if matchchunk(E,ST) = true then put E into field output exit repeat end if end repeat put the milliseconds into M2 put (M2 - M)/1000 into field feedback end mouseUp Each element was created so that the name of that element is the first word of the element - meaning I do not have to keep track of which element we are dealing with - as long as I have the value of the element, I automatically have the name of the element as well. If the word being searched for is in the 500,000th element of the custom property set, then it takes 1.059 seconds to find (with like 4 other programs running on my computer at the same time) If the elements are already combined into a single variable, then lineoffset takes just as long! I guess repeat for each is exceedingly fast. If I could figure out how to use filter on an array - and how to get it to work in the same way as using contains then I would test it with the filter command - but so far this eludes me. In my experiements with this, it seems that an if-then structure was faster than a switch structure - is this always true? ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
Re: custom property searching speed question
Lynch, Jonathan wrote: More tinkering and testing of timing... I changed the customproperty set so that it contains 500,000 records with 3 items in each record. Item 1, the name of the element Item 2, a bunch of words Item 3, a number (100) I created a script that gets the sum of column 3 of this data set - that is, it adds up item 3 for each element. This script: on mouseUp put the milliseconds into M put the customproperties of field theData into myArray put 0 into tSum set the itemdelimiter to numtochar(30) repeat for each element E in myArray add item 3 of E to tSum end repeat put tSum into field output put ((the milliseconds)-M)/1000 into field feedback end mouseUp produces the correct result in .486 seconds! Half a second to sum a column of 500,000 items! What kind of hardware are you using? Tests like that usually take me at least 2 seconds on my PBG4/1KHz. Have you tried looping through lines in a combined string? If your results are like mine you'll shave another 20% off. Yep, kinda hard to beat the performance of a RAM-based database when you have the RAM to architect like that... -- Richard Gaskin Fourth World Media Corporation ___ [EMAIL PROTECTED] http://www.FourthWorld.com ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
Re: custom property searching speed question
Lynch, Jonathan wrote: Anyway, I experimented with a customproperty set created to contain 500,000 elements - each element contains like 7 words. Only the very last element contains the word nonstandard So far, the following script is fastest: on mouseUp put the milliseconds into M put field search text into ST put the customproperties of field theData into myArray repeat for each element E in myArray if matchchunk(E,ST) = true then put E into field output exit repeat end if end repeat put the milliseconds into M2 put (M2 - M)/1000 into field feedback end mouseUp Just a warning / reminder : repeat for each element E in myArray will process each and every element of the array - but it will do it in internal (hash value) order. So when you say only the very last element contains ..., you need to be aware that they are not being looked at in the order you might think - i.e. the one containing that word may be looked at much earlier because of the hashing of the key values. So in your original problem statement, you said (something like) and find the first one that contains - first has a fairly loose meaning in the context of an array if order is crucial to you, you may need to take the keys of the array, sort that, and then access the elements by lookup - this will be much, much slower. -- Alex Tweedly http://www.tweedly.net -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.300 / Virus Database: 266.5.3 - Release Date: 01/03/2005 ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
RE: custom property searching speed question
I am on a dell optiplex GX 270
For each line might get faster - but if I can keep my data stored, at
least partly, in custom properties, that would be very helpful. It's the
whole relational concept. Instead of storing text, a given cell could
store something like {Item 4 of the customproperties[120345] of field
myTable1}.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Richard
Gaskin
Sent: Wednesday, March 02, 2005 5:36 PM
To: How to use Revolution
Subject: Re: custom property searching speed question
Lynch, Jonathan wrote:
More tinkering and testing of timing...
I changed the customproperty set so that it contains 500,000 records
with 3 items in each record.
Item 1, the name of the element
Item 2, a bunch of words
Item 3, a number (100)
I created a script that gets the sum of column 3 of this data set -
that
is, it adds up item 3 for each element.
This script:
on mouseUp
put the milliseconds into M
put the customproperties of field theData into myArray
put 0 into tSum
set the itemdelimiter to numtochar(30)
repeat for each element E in myArray
add item 3 of E to tSum
end repeat
put tSum into field output
put ((the milliseconds)-M)/1000 into field feedback
end mouseUp
produces the correct result in .486 seconds!
Half a second to sum a column of 500,000 items!
What kind of hardware are you using? Tests like that usually take me at
least 2 seconds on my PBG4/1KHz.
Have you tried looping through lines in a combined string? If your
results are like mine you'll shave another 20% off.
Yep, kinda hard to beat the performance of a RAM-based database when you
have the RAM to architect like that...
--
Richard Gaskin
Fourth World Media Corporation
___
[EMAIL PROTECTED] http://www.FourthWorld.com
___
use-revolution mailing list
use-revolution@lists.runrev.com
http://lists.runrev.com/mailman/listinfo/use-revolution
___
use-revolution mailing list
use-revolution@lists.runrev.com
http://lists.runrev.com/mailman/listinfo/use-revolution
RE: custom property searching speed question
When I created the custom property set, I set the numberformat to ##. Thus, element 1 is actually named 01. The unsorted list of the keys is not quite in alphanumeric order, because almost. When I put the unsorted list of the keys in the message box, key 50 is indeed the last key. I also tested this with putting the unusual item in element 30, and it took like .75 seconds to return the result. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Alex Tweedly Sent: Wednesday, March 02, 2005 5:58 PM To: How to use Revolution Subject: Re: custom property searching speed question Lynch, Jonathan wrote: Anyway, I experimented with a customproperty set created to contain 500,000 elements - each element contains like 7 words. Only the very last element contains the word nonstandard So far, the following script is fastest: on mouseUp put the milliseconds into M put field search text into ST put the customproperties of field theData into myArray repeat for each element E in myArray if matchchunk(E,ST) = true then put E into field output exit repeat end if end repeat put the milliseconds into M2 put (M2 - M)/1000 into field feedback end mouseUp Just a warning / reminder : repeat for each element E in myArray will process each and every element of the array - but it will do it in internal (hash value) order. So when you say only the very last element contains ..., you need to be aware that they are not being looked at in the order you might think - i.e. the one containing that word may be looked at much earlier because of the hashing of the key values. So in your original problem statement, you said (something like) and find the first one that contains - first has a fairly loose meaning in the context of an array if order is crucial to you, you may need to take the keys of the array, sort that, and then access the elements by lookup - this will be much, much slower. -- Alex Tweedly http://www.tweedly.net -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.300 / Virus Database: 266.5.3 - Release Date: 01/03/2005 ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
Re: custom property searching speed question
Only 2 seconds? I would think about a month. ;) Michael On Mar 2, 2005, at 4:36 PM, Richard Gaskin wrote: PBG4/1KHz. ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
Re: custom property searching speed question
Richard Gaskin wrote: Lynch, Jonathan wrote: More tinkering and testing of timing... I changed the customproperty set so that it contains 500,000 records with 3 items in each record. Item 1, the name of the element Item 2, a bunch of words Item 3, a number (100) I created a script that gets the sum of column 3 of this data set - that is, it adds up item 3 for each element. This script: on mouseUp put the milliseconds into M put the customproperties of field theData into myArray put 0 into tSum set the itemdelimiter to numtochar(30) repeat for each element E in myArray add item 3 of E to tSum end repeat put tSum into field output put ((the milliseconds)-M)/1000 into field feedback end mouseUp produces the correct result in .486 seconds! Half a second to sum a column of 500,000 items! What kind of hardware are you using? Tests like that usually take me at least 2 seconds on my PBG4/1KHz. 1Khz ? That's your problem - most of us are using machines measured in Mhz or Ghz these days. :-) The script below (which creates the data as well, so you can see exactly what it is doing) takes .236 seconds on my laptop (2.8G Pentium 4) Have you tried looping through lines in a combined string? If your results are like mine you'll shave another 20% off. Only saves 4% for me. (And the combine took over a second - 500% of the summation cost !) on mouseUp local M, myArray local i, E, tSum, t put empty into field lockedField put the millisecs into M repeat with i = 1 to 50 put a i into t put t ,b c d,2 into myArray[t] end repeat put the millisecs-M cr after field lockedField put the millisecs into M put 0 into tSum repeat for each element E in myArray add item 3 of E to tSum end repeat put ((the milliseconds)-M)/1000 tSum cr after field lockedField put the millisecs into M combine myArray with cr put ((the milliseconds)-M)/1000 cr after field lockedField put the millisecs into M put 0 into tSum repeat for each line E in myArray add item 3 of E to tSum --put E cr after field lockedField end repeat put ((the milliseconds)-M)/1000 tSum cr after field lockedField end mouseUp -- Alex Tweedly http://www.tweedly.net -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.300 / Virus Database: 266.5.3 - Release Date: 01/03/2005 ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
Re: custom property searching speed question
Alex Tweedly wrote: Have you tried looping through lines in a combined string? If your results are like mine you'll shave another 20% off. Only saves 4% for me. (And the combine took over a second - 500% of the summation cost !) Of course. In my case I had the option of storing either as a chunk or an array so my measurements included only the actual lookup time, not any of the setup. -- Richard Gaskin Fourth World Media Corporation __ Rev tools and more: http://www.fourthworld.com/rev ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
Re: custom property searching speed question
Alex Tweedly wrote: Have you tried looping through lines in a combined string? If your results are like mine you'll shave another 20% off. Only saves 4% for me. (And the combine took over a second - 500% of the summation cost !) Of course. In my case I had the option of storing either as a chunk or an array so my measurements included only the actual lookup time, not any of the setup. Well... if there's an option of altering the storage might I suggest using three parallel custom property sets. The sum would then only require reading the set for what is currently item 3 into a variable and calling the sum function. I would be rather surprised if this wasn't blindingly fast Cheers Monte ___ use-revolution mailing list use-revolution@lists.runrev.com http://lists.runrev.com/mailman/listinfo/use-revolution
